Skip to content

Releases: pytorch/TensorRT

Release list

Torch-TensorRT v2.14.0

Torch-TensorRT v2.14.0 Latest
Latest

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 10 Sep 05:39
v2.14.0
169cfef
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Torch-TensorRT 2.14.0 Linux x86-64 and Windows targets

PyTorch 2.14, CUDA 12.6/13.0/13.2, TensorRT 11.1, Python 3.10~3.14

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

CUDA 12.6/13.0/13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

aarch64 SBSA Linux and Jetson Thor and Orin:
CUDA 13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

CUDA 13.0/13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

Torch-TensorRT-RTX 2.14.0 Linux x86-64 and Windows targets

PyTorch 2.14, CUDA 12.6/13.0/13.2, TensorRT-RTX 1.6, Python 3.10~3.14

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows:
CUDA 13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

CUDA 12.6/13.0/13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

Torch-TensorRT-Executorch-Runtime 2.14.0 Linux x86-64 and Windows targets

PyTorch 2.14, CUDA 12.6/13.0/13.2, TensorRT 11.1, Python 3.10~3.14

Torch-TensorRT-Executorch-Runtime Wheels are available:

x86-64 Linux:
CUDA 13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

CUDA 12.6/13.0/13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

aarch64 SBSA Linux and Jetson Thor and Orin:
CUDA 13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

CUDA 12.6/13.0/13.2 + Python 3.10-3.14 + Torch 2.14 + TensorRT 11.1

New Features:

Executorch

Torch-TensorRT 2.14 expands its ExecuTorch integration with a more complete export and deployment workflow for TensorRT-accelerated .pte models.

Highlights:

  • New torch_tensorrt.executorch.export() API for producing an inspectable EdgeProgramManager before serialization.
  • Support for exporting multiple named methods into a single .pte artifact, with optional constant methods and ETRecord generation for debugging.
  • torch_tensorrt.save(..., output_format="executorch") remains the streamlined path for one-step model export.
  • Supports composing the TensorRT partitioner with additional ExecuTorch partitioners, enabling mixed TensorRT and CUDA delegation in one artifact.
  • The optional torch-tensorrt-executorch-runtime package enables loading and running exported .pte files through torch_tensorrt.load(..., format="executorch").
  • Continues to use the lightweight native TensorRT ExecuTorch backend, without a Torch or LibTorch dependency at deployment time.
  • Aliased I/O for in-place operations — TensorRT engines can now write directly into a caller-owned or module-owned tensor's storage, so KV-cache updates in streaming and autoregressive inference no longer pay a full cache-sized copy at the engine boundary.
  • Multiple optimization profiles — A single engine can now carry several independently tuned shape profiles selected at runtime by index or automatically, so bimodal workloads like LLM prefill and decode each get kernels tuned for their own shapes instead of splitting the difference.

ExecuTorch support is currently available on Linux(x86-64+aarch64), will Support Windows from 2.15 release.

Multi-Device in TRT-RTX

This release expands multi-GPU support across standard TensorRT and TensorRT-RTX configurations. Native TensorRT collective operations are enabled for TRT-RTX 1.5+ builds on supported Ampere or newer GPUs. Also there is improved process group discovery, including non contiguous subgroup identifiers and serialized save/load, which prevents crashes and ensures distributed engines bind to the correct world group or subgroup.

FP8 Fused Attention Layers / Quantization

This release introduces native FP8 support for TensorRT's IAttention layer. To enable this and improve overall attention layer performance, four attention converters were comprehensively refactored to utilize TensorRT's updated add_attention_v2() API. Beyond core attention support, the update strengthens the FP8 quantization pipeline by adding support for ITensor amax computations, complete with a new export_torch_mode() wrapper to streamline the model export process. Finally, to ensure a smoother out-of-the-box developer experience, this update patches NVIDIA ModelOpt versioning issues within the provided Vision Transformer (ViT) FP8 quantization examples.

Aliased IO and KV Caching

Aliased I/O for in-place operations (#4251)
TensorRT engines can now write in place into a tensor you own, removing the full-tensor copy that in-place operators previously paid at the engine boundary. The main beneficiary is streaming and autoregressive inference with a key/value cache, where each step wrote one timestep but copied the whole cache. Caches passed as inputs and caches registered as module buffers both work, on either runtime, with or without CUDA graphs, and compiled modules still save and load normally. Cases TensorRT can't alias fall back to the previous behavior, so nothing that compiled before stops compiling. This is an ABI-breaking change: previously serialized engines must be rebuilt.

Multi Optimization Profile

A single engine can now carry several independently tuned shape profiles and switch between them at runtime, so one set of weights serves multiple shape regimes without rebuilding. This matters most for autoregressive LLMs, where long prefill and single-token decode previously had to share one tuning point and decode ran on prefill-shaped kernels. Declare profiles with the new profiles argument to torch_tensorrt.Input, then pin one per with block using torch_tensorrt.runtime.optimization_profile, or pass "auto" to select by input shape. Engines that declare no profiles are unaffected.

Improved Complex number lowering in Dynamo

Torch-TensorRT now integrates Pytorch's upstream complex-number decomposition for dynamo compiled models. For torch versions >= 2.14, the new path is enabled automatically and converts complex operations into TensorRT compatible real valued operations while preserving complex I/O and symbolic dynamic shapes.

What's Changed

Read more

Contributors

shoumikhin, narendasan, and 14 other contributors
Assets 41

Torch-TensorRT v2.13.0

Torch-TensorRT v2.13.0

Choose a tag to compare

View all tags
@zewenli98 zewenli98 released this 28 Jul 21:52
v2.13.0
64f0ed4
This commit was signed with the committer’s verified signature.
lanluo-nvidia 岚岚
GPG key ID: FB85EB5ED4A410FE
Verified
Learn about vigilant mode.

Torch-TensorRT 2.13.0 Linux x86-64 and Windows targets

PyTorch 2.13, CUDA 13.0/13.2, TensorRT 11.0, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0/13.2 + Python 3.10-3.13 is Available via PyPI

aarch64 SBSA Linux and Jetson Thor and Orin:
CUDA 13.0/13.2 + Python 3.10–3.13 + Torch 2.13 + TensorRT 11.0

Torch-TensorRT-RTX 2.13.0 Linux x86-64 and Windows targets

PyTorch 2.13, CUDA 13.0/13.2, TensorRT-RTX 1.5, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0/13.2 + Python 3.10-3.13 is Available via PyPI

CUDA 13.0/13.2 + Python 3.10-3.13 is also Available via Pytorch Index

New Features:

GQA/MQA and Decode-Phase Attention Support

Extends Group Query Attention (GQA) and Multi-Query Attention (MQA) alongside traditional Multi-Head Attention (MHA), enabling efficient inference for modern large language models. This release relaxes SDPA validator constraints to allow variable head-count configurations, adds native support for decode-phase attention where query and key sequences differ, and introduces comprehensive HLO-level test coverage across all three attention kernel variants (standard, flash, efficient) with multiple precision levels (fp16, bf16, fp32). These changes unlock optimized inference patterns for LLMs using GQA/MQA while maintaining full backward compatibility with existing MHA-based models.

FP8 Attention Quantization Support

Supports FP8 quantization of the attention in both TensorRT's IAttention path and decomposed attention path (decompose_attention=True). Working with NVIDIA ModelOpt, it detects when attention inputs (Q, K, V) are quantized to FP8 and configures TensorRT's native FP8 MHA kernel for optimal performance on FP8-capable hardware (Hopper and newer GPUs). A ViT quantization example demonstrates both IAttention and decomposed attention paths.

Caller-Selected CUDA Streams and Green Context Support

Enables TensorRT inference on caller-selected CUDA streams across both Torch-TensorRT and ExecuTorch. The PyTorch runtime now respects active non-default streams, while ExecuTorch introduces a scoped CudaStreamGuard API. This supports CUDA Green Contexts, NCCL streams, and custom GPU scheduling without changing default-stream behavior. CUDA Graphs are recaptured when streams change, and completion events safely protect asynchronous execution-context reuse.

TensorRT 11 Support

Torch-TensorRT 2.13 adds support for NVIDIA TensorRT 11, with release packages updated to TensorRT 11.0. The integration has been updated for TensorRT 11 APIs across compilation, runtime execution, packaging, and distributed inference.
Highlights:

  • TensorRT 11.0 support for Python and C++ workflows.
  • Updated CUDA 12 and CUDA 13 package dependencies for TensorRT 11.
  • Updated distributed and NCCL-based operators for TensorRT 11.
  • Improved scaled dot-product attention accuracy using TensorRT 11 fixes.
  • Updated build configurations, CI coverage, and prebuilt release artifacts for TensorRT 11.
    TensorRT 11 is the default supported TensorRT version for Torch-TensorRT 2.13 release packages.

ExecuTorch Support

This release delivers a pure ExecuTorch TensorRT backend for deploying TensorRT-accelerated models in ExecuTorch .pte format. The native runtime path no longer depends on Torch or LibTorch, enabling lightweight C++ deployment using only ExecuTorch and TensorRT runtime components.
Users can continue exporting TensorRT-compiled ExportedProgram and FX models with:
torch_tensorrt.save(model, "model.pte", output_format="executorch")
Highlights:

  • Pure ExecuTorch native backend with no Torch or LibTorch runtime dependency.
  • Support for TensorRT output allocators and data-dependent output shapes.
  • Support for embedding TensorRT engine payloads larger than 2 GiB in .pte files.
  • Target-device selection during ExecuTorch export.
  • Continued support for static-shape and TensorRT profile-based dynamic-shape models.
  • Updated C++ TensorRT ExecuTorch backend and reference runner for Python-free inference.

ExecuTorch Target-Device Metadata

Adds target_device metadata to TensorRT-delegated ExecuTorch partitions so device placement propagates into exported .pte files. Single-engine exports derive the CUDA index from the engine itself, while explicit caller overrides remain supported. This prepares exported programs for CUDA-aware memory planning and direct device-resident delegate buffers.

Caller-Selected CUDA Streams and Green Contexts

Introduces CudaStreamGuard, allowing the ExecuTorch TensorRT delegate to execute on a caller-selected CUDA stream. This enables CUDA Green Contexts, custom GPU scheduling, and asynchronous device-resident execution while preserving existing default behavior.

Large TensorRT Engine Export for ExecuTorch

Supports exporting TensorRT engines larger than 2 GB to ExecuTorch .pte files. Serialized engines are now stored as CPU tensor attributes instead of inline base64 Python string literals, avoiding CPython’s string-size limit during FX graph recompilation. This enables deployment of large models through ExecuTorch without changing the underlying TensorRT engine format.

Unified Python and C++ Runtime with Cross-Runtime Serialization

Unifies the Python and C++ TensorRT runtimes behind TorchTensorRTModule and a common execution operation. Torch-TensorRT now automatically uses the C++ runtime when available and falls back to the pure-Python implementation otherwise (Python-only build). Engine constants are normalized during serialization so models saved under either runtime can be loaded and executed in Python-only or C++-enabled runtime.

Simplified Refit Pipeline

Removes the heuristic cached weight-name mapping previously stored in compiled modules and engine-cache entries. Refit now reconstructs the TensorRT-to-PyTorch weight mapping directly from the new graph on every operation, eliminating stale or incorrect cache matches and the associated fallback logic. Weight refitting and engine caching remain supported, and existing cache entries continue to load with their obsolete mapping metadata ignored.

Multi-Device TensorRT (MD-TRT) Distributed Inference

Adds native multi-device inference built on TensorRT 11's multi-device (MD) engines, enabling TensorRT-accelerated collectives across GPUs driven through standard torch.distributed APIs. This release adds converters for All-to-All, Scatter, and Gather (alongside existing all-reduce, all-gather, and reduce-scatter), and integrates TensorRT's NCCL path with PyTorch's process groups via a distributed_context API that routes each collective to the correct communicator across default and non-default (subgroup) process groups. NCCL library discovery now shares the same libnccl instance PyTorch loads — on TensorRT 11.0+ the runtime loads libnccl.so.2 directly, matching PyTorch's library and removing the need for manual LD_LIBRARY_PATH or symlink workarounds — so distributed, tensor-parallel, and TP-subgroup inference works out of the box.

Shared Dynamic Dimensions Across Inputs

Adds a shared_dims argument to torch_tensorrt.Input for models whose dynamic axes must be equal across multiple inputs (for example, a batch dimension shared by input_ids and attention_mask). Axes given the same name via Input(shared_dims={axis: name}) are exported as a single torch.export.Dim, so the equality constraint is satisfied automatically and torch.export no longer raises a ConstraintViolationError from treating each input's dynamic axis as independent. New shared_dynamic_dims_example documents the end-to-end usage.

What's Changed

Read more

Contributors

shoumikhin, narendasan, and 13 other contributors
Loading

Torch-TensorRT v2.12.1

Torch-TensorRT v2.12.1

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 09 Jun 23:56
v2.12.1
92bb166
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Torch-TensorRT 2.12.1 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT 10.16, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

aarch64 SBSA Linux and Jetson Thor
CUDA 13.0 + Python 3.10–3.13 + Torch 2.12 + TensorRT 10.16

Jetson Orin

  • no torch_tensorrt 2.9/2.10/2.11/2.12 release for Jetson Orin
  • please continue using torch_tensorrt 2.8 release

Torch-TensorRT-RTX 2.12.1 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT-RTX 1.4, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

CUDA 13.0 + Python 3.10-3.13 is also Available via Pytorch Index

What's Changed

  • release 2.12 MD TRT fix by @apbose in #4291
  • fix: cap tensorrt_rtx requirement below 1.5.0 on release/2.12 by @tp5uiuc in #4298
  • fix(lowering): use factored sqrt form for complex acosh rewrite by @tp5uiuc in #4300
  • setup_nccl_for_torch_tensorrt change by @apbose in #4303

Full Changelog: v2.12.0...v2.12.1

Contributors

apbose and tp5uiuc
Loading

Torch-TensorRT v2.12.0

Torch-TensorRT v2.12.0

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 20 May 19:24
v2.12.0
9afefd0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Torch-TensorRT 2.12.0 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT 10.16, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

aarch64 SBSA Linux and Jetson Thor
CUDA 13.0 + Python 3.10–3.13 + Torch 2.12 + TensorRT 10.16

Jetson Orin

  • no torch_tensorrt 2.9/2.10/2.11/2.12 release for Jetson Orin
  • please continue using torch_tensorrt 2.8 release

Torch-TensorRT-RTX 2.12.0 Linux x86-64 and Windows targets

PyTorch 2.12, CUDA 13.0, TensorRT-RTX 1.4, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

CUDA 13.0 + Python 3.10-3.13 is also Available via Pytorch Index

Native Distributed Collectives

In prior versions of Torch-TensorRT, distributed operators were backed by kernels provided by TensorRT-LLM that the user needed to manually install. With Torch-TensorRT 2.12, many of these operations are natively supported which means in deployment, only Torch-TensorRT needs to be installed.

The distributed infrastructure is designed to operate on top of torch.distributed. Once a graph is sharded, traced and compiled. The torch.distributed device mesh can be passed to torch-tensorrt compiled modules using the following API:

trt_model = torch.compile(model, backend="torch_tensorrt", ...)
_ = trt_model(inp)  # warmup — triggers engine build
dist.barrier()

with torch_tensorrt.distributed.distributed_context(dist.group.WORLD, trt_model) as dmodel:
    output = dmodel(inp)

dist.destroy_process_group()
os._exit(0)

This can be done at compile time as well

with torch_tensorrt.distributed.distributed_context(tp_group):
    trt_model = torch.compile(model, backend="torch_tensorrt", ...)
    output = trt_model(inp)

Note: use_distributed_trace is no longer necessary to compile multi-device models, torch-tensorrt will automatically recognize distributed collectives and set the setting for the user.

torchtrtrun

The distributed operations utilize the NCCL version distributed by PyTorch which must be added to LD_PRELOAD before importing torch-tensorrt. As a convience we provide a tool torchtrtrun which is analogous to torchrun that configures these libraries correctly in addition to allowing users to launch models distributed across multiple nodes.

For example:

# Node 0 (rank 0):
torchtrtrun --nproc_per_node=1 --nnodes=2 --node_rank=0 \
  --rdzv_endpoint=<node0-ip>:29500 \
  tensor_parallel_llama_multinode.py

# Node 1 (rank 1):
torchtrtrun --nproc_per_node=1 --nnodes=2 --node_rank=1 \
  --rdzv_endpoint=<node0-ip>:29500 \
  tensor_parallel_llama_multinode.py

Serialization and torch.export

Models sharded and then exported can be compiled and saved to disk before being loaded on a deployment system. By default these modules attempt to bind to the default torch distributed device mesh. If there are multiple valid device meshes availble, the above API can be used to set a specific one to execute the engine.

More information on torch-tensorrt distributed collective support can be found here: https://docs.pytorch.org/TensorRT/tutorials/deployment/distributed_inference.html#multinode-inference

More information on native multi-device collectives can be found here: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/work-with-transformers.html#multi-device-attention-preview-feature

ExecuTorch Support

Torch-TensorRT 2.12 introduces initial ExecuTorch integration for exporting and running TensorRT-accelerated models in ExecuTorch .pte format. Users can now
save TensorRT-compiled ExportedProgram / FX models with:

torch_tensorrt.save(model, "model.pte", output_format="executorch")

This release adds a TensorRT ExecuTorch backend, partitioner, and serialization path that embeds TensorRT engine payloads directly into the .pte using the
same engine metadata format as the Torch-TensorRT runtime. The release package also includes a C++ TensorRT ExecuTorch backend source package and a reference
ExecuTorch runner showing how to load .pte files, initialize the TensorRT delegate, bind inputs/outputs, and execute inference without requiring Python at
runtime.

Highlights:

  • New torch_tensorrt.executorch Python APIs: TensorRTBackend, TensorRTPartitioner, and get_edge_compile_config.
  • New output_format="executorch" save path for generating ExecuTorch .pte models.
  • Support for static-shape and TensorRT profile-based dynamic-shape export examples.
  • New native C++ TensorRT ExecuTorch backend and reference runner included in libtorchtrt.tar.gz.
  • Engines that require TensorRT output allocators, such as data-dependent output shape engines, are not supported yet.
  • In Torch-TensorRT 2.12, the ExecuTorch integration still depends on LibTorch in the native runtime path.
    In the next Torch-TensorRT 2.13 release is planned to move this to a pure ExecuTorch backend implementation without the LibTorch runtime dependency.

Known Limitations

  • ExecuTorch support still depends on the Torch/LibTorch C++ libraries used by Torch-TensorRT; this release does not provide a pure
    ExecuTorch-only TensorRT deployment path.
  • TensorRT engine payloads larger than 2 GiB are not supported when embedded in an ExecuTorch .pte file.
  • Selecting a target device during ExecuTorch export is not currently supported. Exported .pte files default to cuda:0.

Comprehensive Attention Support

This release extends the TRT attention converters to support GQA/MQA and decode-phase attention based on TensorRT IAttentionLayer. Specifically, it covers all SDPA kernel variants, MHA/GQA/MQA attention patterns, causal vs non-causal masking, bool/float/broadcast mask shapes, decode-phase attention (seq_q=1), non-power-of-2 head dims, LLM-realistic configs, and multiple dtypes. This feature is enabled by default. If you want to turn it off, please set decompose_attention=True.

Known Limitations

  • TensorRT 10.x (will be resolved in TRT 11.0) and TensorRT-RTX-1.4:
    For TensorRT 10.x, large causal sequences of k/v (seq >= 512, is_causal=True) in FP16/BF16
    IAttentionLayer produces ~80% element mismatch at long sequences. Thus, we use FP32 for
    the scale factor. If you want to use the accurate dtype, please set decompose_attention=True
    or upgrade to TRT 11.0 or later.

Comprehensive Complex Numerics Support

Torch-TensorRT can now compile models containing complex64/complex128 tensors end-to-end. TensorRT itself has no native complex dtype — a lowering pass intercepts complex subgraphs before partitioning and rewrites them into equivalent real arithmetic on a (..., 2) last-dim layout (real/imag interleaved), so the engine only sees standard float ops and callers don't have to change anything.

This unlocks compilation of models that use complex arithmetic for rotary position embeddings — Llama 3 (1D RoPE), and video generation transformers like CogVideoX, Wan, and HunyuanVideo (3D RoPE) — including under dynamic shapes and in distributed (multi-GPU) settings.

What's supported

  • Complex inputs (placeholders) and buffers (get_attr) are rewritten to real-valued equivalents. placeholder(complex64) becomes placeholder(float32) with an appended trailing dim of 2; complex buffers are replaced via torch.stack([t.real, t.imag], dim=-1). Dynamic-shape SymInts are preserved across the rewrite.
  • Complex multiply (aten.mul.Tensor between two complex operands) is decomposed to the standard identity (ac − bd) + (ad + bc)i.
  • view_as_complex / view_as_real are erased — they become identities once the layout is already (..., 2).
  • Shape-manipulation ops are handled with the trailing real/imag dim in mind: reshape / view / _unsafe_view, flatten, unsqueeze, squeeze, permute, transpose, t, cat, stack, select, slice, narrow, roll, flip, split, chunk, expand, repeat. Negative dim indices are auto-shifted by −1 so dim=-1 keeps meaning "the original last complex dim."
  • Math ops over complex inputs — mm / bmm / matmul (including complex×real and real×complex mixed forms), abs, angle, real, imag, sin/cos/exp/log on complex, reciprocal (for scalar / complex), sum.dim_IntList / mean.dim / prod.dim_int, ones_like / full_like (correctly initialise as 1+0i / fill+0i).
  • Engine outputs that stay complex — models that return a complex tensor without going through view_as_real are also detected via a forward scan that collects unbounded complex nodes.
  • Runtime I/O — complex inputs can be passed as-is at call time. The runtime modules automatically apply torch.view_as_real(x).contiguous() to complex inputs before handing them to the engine, and rebuild complex outputs on the way back.
  • truncate_double=True lowers complex128 to float32 (vs the default float64)
  • Refit caching — complex buffers via a last-dim slice-matching stage in _save_weight_mapping, plus a tuple-keyed (sd_key, last_dim, idx) lookup in construct_refit_mapping_from_weight_name_map. Verification picks the real-unpacked branch for the reference module when complex inputs are present.
  • Graceful fallback — complex ops the rewriter doesn't know how to handle now fall back to PyTorch execution rather than failing the compile.

Known limitations

  • Only view_as_real-anchored subgraphs and forward-scanned comple...
Read more

Contributors

narendasan, SandSnip3r, and 8 other contributors
Loading

Torch-TensorRT v2.11.0

Torch-TensorRT v2.11.0

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 07 Apr 17:03
v2.11.0
0cc00aa
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Torch-TensorRT 2.11.0 Linux x86-64 and Windows targets

PyTorch 2.11, CUDA 12.6 12.8 12.9, 13.0, TensorRT 10.15, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

CUDA 12.6/12.8/12.9/13.0 + Python 3.10-3.13 is also Available via Pytorch Index

aarch64 SBSA Linux and Jetson Thor
CUDA 13.0 + Python 3.10–3.13 + Torch 2.11 + TensorRT 10.15

Jetson Orin

  • no torch_tensorrt 2.9/2.10/2.11 release for Jetson Orin
  • please continue using torch_tensorrt 2.8 release

Torch-TensorRT-RTX 2.11.0 Linux x86-64 and Windows targets

PyTorch 2.11, CUDA 12.9, 13.0, TensorRT-RTX 1.3, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

CUDA 12.9/13.0 + Python 3.10-3.13 is also Available via Pytorch Index

note: tensorrt-rtx 1.3 wheel is not in pypi yet, so please download the tarball from https://developer.nvidia.com/tensorrt-rtx
install the wheel from tarball

IAttention Layer

In this release, TensorRT's native IAttention layer is used by default to handle various attention-related ATen ops, including SDPA, Flash-SDPA, Efficient-SDPA, and cuDNN-SDPA. This integration enables more efficient execution and can improve model performance. To explicitly enable this behavior, set decompose_attention=False in the compile() function. When enabled, the native TensorRT implementation is utilized to achieve optimized attention computation.
However, due to current TensorRT limitations, certain operations such as compute_log_sumexp and Grouped Query Attention (GQA) are not yet supported. If these cases are encountered, an informational prompt will be displayed during compilation. Alternatively, you can set decompose_attention=True to decompose the attention ops into multiple basic ATen ops. Although this approach may not achieve the same level of performance optimization, it offers broader operator coverage and greater compatibility across different model architectures.

Improvements to the Symbolic Shape System

Two key improvements have been made to the symbolic shape system used to track mutations on dynamic dimensions through out the body of the graph.

  1. A shape prop formula is recorded as metadata in for every compiled engine.

Now instead of needing to instantiate the engine in order to do key tasks like serialization, retracing or similar tasks which require fake tensor propagation, we record the shape relation between inputs and outputs for every TRT subgraph at compile time and store it as metadata. The torch.ops.tensorrt.execute_engine meta kernel now just replays this function in the new shape environment.
This should enable more seamless integration with the rest of the torch.compile ecosystem as meta kernels in Torch-TensorRT will now work in the same way as other meta kernels.

  1. For unbounded shape ranges, we now select sane defaults

Dynamic shapes are lazily inserted into Dynamo graphs. This is most noticeable when using Torch-TensorRT as a backend for torch.compile. Here when the first inference call is made to a boxed function

trt_mod = torch.compile(mod, backend="tensorrt")
trt_mod(*inputs)

The shapes of intermediate tensors are considered static and derived eagerly from the shapes of input tensors.

If input shapes then change

trt_mod(*other_sized_inputs)

TorchDynamo will start marking dimensions where shapes differ as dynamic dimensions. However, it does not assume upper bounds and critically, has no "optimal" or target size as required by TensorRT.

As such when we see such [FIXED_SIZE, inf) ranges, we set a sane upper-bound (max_int / max_dims) and assume the optimal size as the mid point of that range. We highly recommend users explicitly set dynamic shape bounds for both torch.export and torch.compile use cases, but this system can serve as fallback. https://docs.pytorch.org/TensorRT/user_guide/compilation/dynamic_shapes.html

Torch-TensorRT-RTX

TensorRT-RTX is a JIT focused version of TensorRT that allows users to target many different hardware platforms with one artifact. It allows developers to easily provide performance to their users across the many variations of RTX GPUs. In previous versions of Torch-TensorRT we have provided source code support for using TensorRT-RTX as a backend for Torch-TensorRT, allowing users to get access to the same workflows they would with standard Torch-TensorRT with more JIT oriented optimization approach.

With 2.11 Torch-TensorRT-RTX has graduated to its own package that you can install with pip install torch-tensorrt-rtx. This package uses all the same APIs as Torch-TensorRT, just with a different backend. torch-tensorrt-rtx 2.11 targets TensorRT-RTX 1.3. For 2.11, TensorRT-RTX must be installed via a wheel distributed on developer.nvidia.com: https://developer.nvidia.com/tensorrt-rtx

Know Limitations:

  • bf16 precision is generally supported, however it is possible in some models that there may still be numerical accuracy issues. This will be addressed in futuer versions of TensorRT-RTX
  • There is a know accuracy issue when running Grouped Query Attention TensorRT-RTX which will be addressed in a future release of TensorRT-RTX

run_llm int8 quantization

We have added support for performing post training quantization in int8 precision from the command line using the run_llm tool .
You can apply int8 quantization backed by the TensorRT-Model-Optimizer-Toolkit using --quant_format fp8

python run_llm.py --model meta-llama/Llama-3.1-8B --quant_format fp8 --prompt "What is parallel programming?" --model_precision FP16 --num_tokens 128

Empty Tensor

The Torch-TensorRT Runtime has

We have added support for providing torch empty tensors (tensors with one or more zero sized dimensions) as input to Torch-TensorRT compiled programs.

Under the hood we use TensorRT native empty tensor semantics. Empty tensors are marked by a 1B placeholder input to the engine. Both the python and C++ runtimes support this feature

What's Changed

Read more

Contributors

narendasan, SandSnip3r, and 6 other contributors
Loading

Torch-TensorRT v2.10.0

Torch-TensorRT v2.10.0

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 20 Feb 02:13
v2.10.0
1a1c2e2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Torch-TensorRT 2.10.0 Linux x86-64 and Windows targets

PyTorch 2.10, CUDA 12.9, 13.0, TensorRT 10.14, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0 + Python 3.10-3.13 is Available via PyPI

CUDA 12.9/13.0 + Python 3.10-3.13 is also Available via Pytorch Index

aarch64 SBSA Linux and Jetson Thor
CUDA 13.0 + Python 3.10–3.13 + Torch 2.10 + TensorRT 10.14

Jetson Orin

  • no torch_tensorrt 2.9/2.10 release for Jetson Orin
  • please continue using torch_tensorrt 2.8 release

Important Changes

Retracing is enabled as the default behavior of saving a compiled graph module with torch_tensorrt.save. Torch-TensorRT re-exports the graph using torch.export.export(strict=False) to save it. This preserves the completeness of the output FX Graph and fills in metadata.

New Features

LLM improvements

The run_llm script now supports compiling models that have previously been quantized using the TensorRT Model Optimizer
Toolkit and uploaded to HuggingFace.

Now we support the following inference scenarios:

Standard high precision model, directly compile and run inference in fp16/bf16 via torch_tensorrt Autocast

python run_llm.py --model Qwen/Qwen2.5-0.5B-Instruct \
--prompt "What is parallel programming?" \
--model_precision FP16 --num_tokens 128 \
--cache static_v2 --enable_pytorch_run

Standard high precision model, use TensorRT Model optimizer to quantize and compile on device and then run inference in fp8/nvfp4 precision

python run_llm.py --model Qwen/Qwen2.5-0.5B-Instruct  \
--prompt "What is parallel programming?"  \
--model_precision FP16 \
--quant_format fp8 --num_tokens 128 \
--cache static_v2 --enable_pytorch_run

Previously quantized model uploaded to Huggingface, directly compile and run inference infp8/nvfp4

python run_llm.py --model nvidia/Qwen3-8B-FP8 \
--prompt "What is parallel programming?"  \
--model_precision FP16 \
--quant_format fp8 \
--num_tokens 128 \
--cache static_v2 --enable_pytorch_run

Notes:
--model_precision
this is mandatory, it is used to tell llm tool what is the model's precision
--quant_format
this is optional, it is only used for quantized model inference
for the pre-quantized modelopt checkpoint, this is to tell

Improvements to Engine Caching

Before this release, since weight-stripped engines can be refitted only once due to the limitation of TensorRT (<10.14), we cached weighted engines to make sure Engine Caching feature work properly, which occupied unnecessary hard disk. Since this release, if users install TensorRT >= 10.14, engine caching will only save weight-stripped engines on disk regardless of compilation_settings.strip_engine_weights, and then, when users pull out the cached engine, it will be automatically refitted and kept refittable all the time, which means compiled TRT modules can be refitted multiple times with the function refit_module_weights(). e.g.:

for _ in range(3):
    trt_gm = refit_module_weights(trt_gm, exp_program)

Autocast

Before TensorRT 10.12, TensorRT would implicitly pick kernels for layers that result in the best performance (i.e., weak typing). Weak typing behavior is deprecated in newer TensorRT versions, but it is a good way to maximize performance. Therefore, in this release, we want to provide a solution for users to enable mixed precision behavior like weak typing, which is called Autocast.

Unlike PyTorch Autocast, Torch-TensorRT Autocast is a rule-based autocast, which intelligently selects nodes to
keep in FP32 precision to maintain model accuracy while benefiting from reduced precision on the rest of the nodes.
Torch-TensorRT Autocast also supports users to specify which nodes to exclude from Autocast, considering some nodes
might be more sensitive to affecting accuracy. In addition, Torch-TensorRT Autocast can cooperate with PyTorch Autocast,
allowing users to use both PyTorch Autocast and Torch-TensorRT Autocast in the same model. Torch-TensorRT Autocast
respects the precision of the nodes within PyTorch Autocast context. Please refer to Torch-TRT mixed precision doc for more details.

Compilation Resource Management

Compiling large models on limited-resource hardware is challenging. Before this release, to successfully compile the FLUX model (24GB), we needed at least 128GB of host memory, which is >5x of the model size. This huge consumption limited Torch-TensorRT's capability to compile large models with limited resources.

Host Memory Optimization

Introduce the feature of trimming malloc memory, thus reducing peak host memory consumption.
bash export TORCHTRT_ENABLE_BUILDER_MALLOC_TRIM=1 python example.py
By using the environment variable, the peak memory usage can be reduced to 3x.

If the cuda memory is sufficient, you can disable by setting offload_module_to_cpu=False to further reduce the host memory to 2x. More detailed explanation can be found here: https://github.com/pytorch/TensorRT/blob/main/docsrc/contributors/resource_management.rst

Resource Aware Partitioner

A new feature called Resource Aware Partitioner is introduced to address situations where the available host memory is smaller than 3x of the model size. In compilation settings, set enable_resource_partitioning=True and (optionally) set a cpu_memory_budget, the partitioner will automatically shard the graph such that the compilation resource consumption can fit into very constrained resources (<2x) without sacrificing performance and accuracy.
Example usage can be found here:
https://github.com/pytorch/TensorRT/blob/b7ae84fc020b1f0428b019d39c6284c7d52626e7/examples/dynamo/low_cpu_memory_compilation.py

Debugger

TensorRT API Capture

In this release, we have added the TensorRT API Capture and Replay feature
which streamlines the process of reproducing and debugging issues within your model.
It allows you to record the engine-building phase of your model and later replay the engine-build steps.

Capture:
The capture feature is by default disabled.
You can enable the capture feature via environment variable: TORCHTRT_ENABLE_TENSORRT_API_CAPTURE=1
TORCHTRT_ENABLE_TENSORRT_API_CAPTURE=1 python your_model_test.py
You should see the shim.json and shim.bin generated after enable the capture.

Replay:
Use tensorrt_player tool to replay the captured trt engine build without the original framework
tensorrt_player -j /absolute/path/to/shim.json -o /absolute/path/to/output_engine

Limitations:
-This feature is currently restricted to Linux(x86-64 and aarch64) only.
-This feature is currently restricted to capture and record 1 trt engine only,
in case you have graph break, there are multiple engines are built, only the first engine is recorded.
In the next release we will support multiple engines are all recorded in the same bin file.

You can see more details in
https://docs.pytorch.org/TensorRT/getting_started/capture_and_replay.html?highlight=capture+replay#

What's Changed

Read more

Contributors

narendasan, peri044, and 8 other contributors
Loading
0 Join discussion

Torch-TensorRT v2.9.0

Torch-TensorRT v2.9.0

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 17 Oct 15:58
v2.9.0
8767d9b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

PyTorch 2.9, CUDA 13.0 TensorRT 10.13, Python 3.13

Torch-TensorRT 2.9.0 Linux x86-64 and Windows targets PyTorch 2.9, TensorRT 10.13, CUDA 13.0, 12.8, 12.6 and Python 3.10 ~ 3.13

Python

x86-64 Linux and Windows

aarch64 SBSA Linux and Jetson Thor

NOTE: You must explicitly install TensorRT or use system installed TensorRT wheels for aarch64 platforms

uv pip install torch torch-tensorrt tensorrt

aarch64 Jetson Orin

 - no torch_tensorrt 2.9 release for Jetson Orin, please continue using torch_tensorrt 2.8 release

C++

x86-64 Linux and Windows

  • CUDA 13.0 Tarball / Zip

Deprecations

FX Frontend

The FX frontend was the precursor to the Dynamo frontend and a number of Dynamo components were shared between the two. Now that the Dynamo frontend is stable and all shared components have been decoupled we will no longer ship the FX frontend in binary releases starting in H1Y26. The FX frontend will remain in the source tree for the foreseeable future so source builds can reinstall the frontend if necessary.

New Features

LLM and VLM improvements

In this release, we’ve introduced several key enhancements:

  • Sliding Window Attention in SDPA Converter : Added support for sliding window attention, enabling successful compilation of the Gemma3 model (Gemma3-1B).
  • Dynamic Custom Lowering Passes
    Refactored the lowering framework to allow users to dynamically register custom passes based on the configuration of Hugging Face models.
  • Vision-Language Model (VLM) Support
    • Added support for Eagle2 and Qwen2.5-VL models via the new run_vlm.py utility.
    • run_vlm.py enables compilation of both vision and language components of a VLM model. It also supports KV caching for efficient VLM generation.

See the documentation for detailed instructions on running these models.

TensorRT-RTX

TensorRT-RTX is a JIT-first version of TensorRT. Where as TensorRT will perform tactic selection and fusions during a build phase. TensorRT-RTX allows you to distribute builds prior to specializing for specific hardware so that one GPU agnostic package can be distributed to all users of your builds. Then on first use, TensorRT RTX will tune for the specific hardware your users are running. Torch-TensorRT-RTX is a build of Torch-TensorRT that uses the TensorRT-RTX compiler stack inplace of standard TensorRT. All APIs are identical to Torch-TensorRT, however, some features such as weak-typing and at compile time post training quantization are not supported.

Improvements

  • Closed a number of performance gaps between Torch-TensorRT and ONNX TensorRT constructed graphs

What's Changed

Read more

Contributors

HolyWu, peri044, and 7 other contributors
Loading

Torch-TensorRT v2.8.0

Torch-TensorRT v2.8.0

Choose a tag to compare

View all tags
@narendasan narendasan released this 09 Aug 06:02
v2.8.0
e94d48c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

PyTorch 2.8, CUDA 12.8 TensorRT 10.12, Python 3.13

Torch-TensorRT 2.8.0 Standard Linux x86-64 and Windows targets PyTorch 2.8, TensorRT 10.12, CUDA 12.6, 12.8, 12.9 and Python 3.9 ~ 3.13

Platform support

In addition to the standard Windows x86-64 and Linux x86-64 releases, we now provide binary builds for SBSA and Jetson:

Deprecations

New Features

AOT-Inductor Pythonless Deployment

Stability: Beta

Historically TorchScript has been used to run Torch-TensorRT programs outside of a Python interpreter. Both the dynamo/torch.compile frontend and the TorchScript frontends supported this TorchScript deployment workflow.

Old
trt_model = torch_tensorrt.compile(model, ir="dynamo", arg_inputs=[...])
ts_model = torch.jit.trace(trt_model, inputs=[...])
ts_model.save("trt_model.ts")

Now you can achieve a similar result using AOT-Inductor. AOTInductor is a specialized version of TorchInductor, designed to process exported PyTorch models, optimize them, and produce shared libraries as well as other relevant artifacts. These compiled artifacts are specifically crafted for deployment in non-Python environments.

Torch-TensorRT can embed TensorRT engines in AOTInductor libraries to accelerate models further. You are also able to combine Inductor kernels with TensorRT engines via this method. This allows users to deploy their models outside of Python using torch-compile native technologies.

New
with torch.no_grad():
    cg_trt_module = torch_tensorrt.compile(model, **compile_settings)
    torch_tensorrt.save(
        cg_trt_module,
        file_path=os.path.join(os.getcwd(), "model.pt2"),
        output_format="aot_inductor",
        retrace=True,
        arg_inputs=example_inputs,
    )

This model.pt2 file can then be loaded in either Python or C++ using Torch APIs.

import torch
import torch_tensorrt
model = torch._inductor.aoti_load_package(os.path.join(os.getcwd(), "model.pt2"))
#include <iostream>
#include <vector>

#include "torch/torch.h"
#include "torch/csrc/inductor/aoti_package/model_package_loader.h"

int main(int argc, const char* argv[]) {

  std::string trt_aoti_module_path = "model.pt2";
  c10::InferenceMode mode;

  torch::inductor::AOTIModelPackageLoader loader(trt_aoti_module_path);
  std::vector<torch::Tensor> inputs = {torch::randn({8, 10}, at::kCUDA)};
  std::vector<torch::Tensor> outputs = loader.run(inputs);
  std::cout << "Result from the first inference:"<< std::endl;
  std::cout << outputs << std::endl;

  return 0;
}

More information can be found here https://docs.pytorch.org/TensorRT/user_guide/runtime.html as we as a code example here: https://github.com/pytorch/TensorRT/blob/release/2.8/examples/torchtrt_aoti_example/inference.cpp

PTX Plugins

Stability: Stable

In Torch-TensorRT 2.7.0 we introduced auto-generated plugins which allows users to automatically wrap kernels / PyTorch custom Operators into TensorRT plugins to run their models without a graph break. In 2.8.0 we extend this system to support PTX based plugins which enables users to serialize and run their TensorRT engines without requiring any PyTorch / Triton / Python in the runtime or access to the original kernel implementation. This approach also has the added benefit of lower overhead than the auto-generated plugin system for achieving maximum performance.

Example below shows how to register a custom operator, generate the necessary plugin, and integrate it into the TensorRT execution graph. [the example]
(https://github.com/pytorch/TensorRT/blob/main/examples/dynamo/aot_plugin.py)

Hierarchical Multi-backend Adjacency Partitioner

Stability: Experimental

The Hierarchical Multi-backend Adjacency Partitioner enables sophisticated model partitioning strategies for distributing PyTorch models across multiple backends based on operator support and priority ordering. A prototype partitioner has been added to the package which allows graphs to be split across multiple backends (e.g., TensorRT, PyTorch Inductor) based on operator capabilities. By providing a backend preference order operators are assigned to the highest-priority backend that supports them.

Please refer to the example for usage.

Model Optimizer-Based NVFP4 Quantization (PTQ) Support for Linux

Stability: Stable

Introducing NVFP4 for efficient and accurate low-precision inference on the Blackwell GPU architecture.
Currently, the workflow supports quantizing models from FP16 → NVFP4.

Directly quantizing from FP32 → NVFP4 is not recommended as it may lead to accuracy degradation. Instead, first convert or train the model in FP16, then quantize to NVFP4.

Full example:
https://github.com/pytorch/TensorRT/blob/release/2.8/examples/apps/flux_demo.py

run_llm and KV Caching

Stability: Beta

We’ve introduced a KV caching implementation for Torch-TensorRT using native TensorRT operations, yielding significant improvements in inference performance for autoregressive large language models (LLMs). KV caching is a crucial optimization that reduces latency by reusing attention activations across decoding steps. In our approach, the KV cache is modeled as fixed-size tensor inputs and outputs, with outputs from each decoding step looped back as inputs to update the cache incrementally. This update is performed using TensorRT-supported operations such as slice, concat, and pad. The design allows step-wise cache updates while preserving compatibility with TensorRT’s optimization workflow and engine serialization.

We’ve also introduced a new utility, run_llm.py, to run inference on popular LLMs with KV caching enabled.

To run a Qwen3 model using KV caching with Torch-TensorRT, use the following command:

python run_llm.py --model Qwen/Qwen3-8B --prompt "What is parallel programming?" --precision FP16 --num_tokens 128 --cache static_v2 --benchmark

Please refer to Compiling LLM models from Huggingface for more details and limitations.

Debugger

We introduced a new debugger for better usability and a debugging experience for Torch-TensorRT. The debugger centralized all debugger settings, such as logging level from critical to info, and engine profiling. We also introduced fx graph visualization in the debugger, where you can specify the specific lowering pass before/which you want to draw the graph. Moreover, the debugger can provide engine profiling and layer information that is compatible with TREX, an engine visualization tool developed by TensorRT, that better explains the engine structure.

Model Zoo

We have expanded support to include several popular models from the Qwen3 and Llama3 series. In this release, we’ve also addressed various performance and accuracy issues to improve overall stability. For a complete list of supported models, please refer to the Supported Models section.

Bug Fixes

Refit

Refit has been re-enabled for Python 3.13 after being disabled in 2.7.0

  • Reduced memory overhead by offloading model to CPU

Performance improvements

  • Linear converter was reverted to the earlier implementation because it shows perf improvements in fp16 on some models (e.g., BERT)
  • Group Norm converter was simplified to reduce unnecessary TensorRT ILayers
  • The constants in the BatchNorm converter are now folded at compile time, leading to significant performance improvements.
  • SDPA op decomposition is optimized, resulting in same or better performance as ONNX-TensorRT for transformer-based diffusion models such as Stable Diffusion 3/WAN2.1/FLUX

What's Changed

Read more

Contributors

narendasan, HolyWu, and 10 other contributors
Loading

Torch-TensorRT v2.6.1

Torch-TensorRT v2.6.1

Choose a tag to compare

View all tags
@lanluo-nvidia lanluo-nvidia released this 03 Jun 04:49
v2.6.1
707f16d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

Full Changelog: v2.6.0...v2.6.1

Contributors

lanluo-nvidia
Loading

Torch-TensorRT v2.7.0

Torch-TensorRT v2.7.0

Choose a tag to compare

View all tags
@narendasan narendasan released this 07 May 23:44
v2.7.0
8250319
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

PyTorch 2.7, CUDA 12.8, TensorRT 10.9, Python 3.13

Torch-TensorRT 2.7.0 targets PyTorch 2.7, TensorRT 10.9, and CUDA 12.8, (builds for CUDA 11.8/12.4 are available via the PyTorch package index - https://download.pytorch.org/whl/cu118 https://download.pytorch.org/whl/cu124). Python versions from 3.9-3.13 are supported. We no longer provide builds for the pre-cxx11-abi, all wheels and tarballs will use the cxx11 ABI.

Known Issues

  • Engine refitting is disabled in Python 3.13.

Using Self Defined Kernels in TensorRT Engines using Automatic Plugin Generation

Users may develop their own custom kernels using DSLs such as OpenAI Triton. Through the use of PyTorch Custom Ops and Torch-TensorRT Automatic Plugin Generation, these kernels can be called within the TensorRT engine with minimal extra code required.

@triton.jit
def elementwise_scale_mul_kernel(X, Y, Z, a, b, BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(0)
    # Compute the range of elements that this thread block will work on
    block_start = pid * BLOCK_SIZE
    # Range of indices this thread will handle
    offsets = block_start + tl.arange(0, BLOCK_SIZE)
    # Load elements from the X and Y tensors
    x_vals = tl.load(X + offsets)
    y_vals = tl.load(Y + offsets)
    # Perform the element-wise multiplication
    z_vals = x_vals * y_vals * a + b
    # Store the result in Z
    tl.store(Z + offsets, z_vals)


@torch.library.custom_op("torchtrt_ex::elementwise_scale_mul", mutates_args=())  # type: ignore[misc]
def elementwise_scale_mul(
    X: torch.Tensor, Y: torch.Tensor, b: float = 0.2, a: int = 2
) -> torch.Tensor:
    # Ensure the tensors are on the GPU
    assert X.is_cuda and Y.is_cuda, "Tensors must be on CUDA device."
    assert X.shape == Y.shape, "Tensors must have the same shape."

    # Create output tensor
    Z = torch.empty_like(X)

    # Define block size
    BLOCK_SIZE = 1024

    # Grid of programs
    grid = lambda meta: (X.numel() // meta["BLOCK_SIZE"],)

    # Launch the kernel with parameters a and b
    elementwise_scale_mul_kernel[grid](X, Y, Z, a, b, BLOCK_SIZE=BLOCK_SIZE)

    return Z

@torch.library.register_fake("torchtrt_ex::elementwise_scale_mul")
def _(x: torch.Tensor, y: torch.Tensor, b: float = 0.2, a: int = 2) -> torch.Tensor:
    return x

torch_tensorrt.dynamo.conversion.plugins.custom_op("torchtrt_ex::elementwise_scale_mul", supports_dynamic_shapes=True, requires_output_allocator=False)

trt_mod_w_kernel = torch_tensorrt.compile(module, ...)

torch_tensorrt.dynamo.conversion.plugins.custom_op will generate a TensorRT plugin using the Quick Deploy Plugin system and using PyTorch's FakeTensor mode by reusing information required to register a Torch custom op to use with TorchDynamo. It will also generate the Torch-TensorRT converter to insert the plugin to the TensorRT engine.

QDP Plugins for Torch Custom Ops and Converters for QDP Plugins can be generated individually using

torch_tensorrt.dynamo.conversion.plugins.generate_plugin(
    "torchtrt_ex::elementwise_scale_mul"
)
torch_tensorrt.dynamo.conversion.plugins.generate_plugin_converter(
    "torchtrt_ex::elementwise_scale_mul",
    supports_dynamic_shapes=True,
    requires_output_allocator=False,
)

MutableTorchTensorRTModule improvements

MutableTorchTensorRTModule automatically recompiles if the engine becomes invalid. Previously, engines would assume static shape which means that if a user provides a different sized input, the graph would recompile or pull from engine cache. Now developers are able to provide shape hints to the MutableTorchTensorRTModule which will allow the module to handle a broader range of inputs without recompiling. For example:

pipe.unet = torch_tensorrrt.MutableTorchTensorRTModule(pipe.unet, **settings)
BATCH = torch.export.Dim("BATCH", min=2, max=24)
_HEIGHT = torch.export.Dim("_HEIGHT", min=16, max=32)
_WIDTH = torch.export.Dim("_WIDTH", min=16, max=32)
HEIGHT = 4 * _HEIGHT
WIDTH = 4 * _WIDTH
args_dynamic_shapes = ({0: BATCH, 2: HEIGHT, 3: WIDTH}, {})
kwargs_dynamic_shapes = {
    "encoder_hidden_states": {0: BATCH},
    "added_cond_kwargs": {
        "text_embeds": {0: BATCH},
        "time_ids": {0: BATCH},
     },
     "return_dict": None,
}
pipe.unet.set_expected_dynamic_shape_range(
    args_dynamic_shapes, kwargs_dynamic_shapes
)

Data Dependent Shape support

For networks that produce outputs whose shapes are dependent on the shape of the input, the output buffer must be allocated at runtime. To support this use case we have added a new runtime mode Dynamic Output Allocation Mode to support Data Dependent Shape (DDS) operations, such as NonZero op. (#3388)

Note:
  • Dynamic output allocation mode cannot be used in conjunction with CUDA Graphs nor pre-allocated outputs feature.
  • Without dynamic output allocation, the output buffer is allocated based on the inferred output shape based on input size.

There are two scenarios in which dynamic output allocation is enabled:

  1. The model has been identified at compile time to require dynamic output allocation for at least one TensorRT subgraph. These models will engage the runtime mode automatically (with logging) and are incompatible with other runtime modes such as CUDA Graphs. Converters can declare that subgraphs that they produce will require the output allocator using requires_output_allocator=True there by forcing any model which utilizes the converter to automatically use the output allocator runtime mode. e.g.,
    @dynamo_tensorrt_converter(
        torch.ops.aten.nonzero.default,
        supports_dynamic_shapes=True,
        requires_output_allocator=True,
    )
    def aten_ops_nonzero(
        ctx: ConversionContext,
        target: Target,
        args: Tuple[Argument, ...],
        kwargs: Dict[str, Argument],
        name: str,
    ) -> Union[TRTTensor, Sequence[TRTTensor]]:
        ...
  1. Users may manually enable dynamic output allocation mode via the torch_tensorrt.runtime.enable_output_allocator context manager.
    # Enables Dynamic Output Allocation Mode, then resets the mode to its prior setting
    with torch_tensorrt.runtime.enable_output_allocator(trt_module):
        ...

Tiling Optimization support

Tiling optimization enables cross-kernel tiled inference. This technique leverages on-chip caching for continuous kernels in addition to kernel-level tiling. It can significantly enhance performance on platforms constrained by memory bandwidth. (#3444)

We currently support four tiling strategies "none", "fast", "moderate", "full". A higher level allows TensorRT to spend more time searching for better tiling strategy. Here's an example to call tiling optimization:

    compiled_model = torch_tensorrt.compile(
        model,
        ir="dynamo",
        inputs=inputs,
        tiling_optimization_level="full",
        l2_limit_for_tiling=10,
    )

Model Zoo additions

  • Added support for compiling the FLUX.1-dev 12B model in our model zoo. An example is available here. Quantized variants of FLUX are under development as part of future work.

General Improvements

  • Improved BF16 support in model compilation by fixing bugs and adding new tests to cover both full-graph and graph-break scenarios.
  • Significantly accelerated model compilation time (#3396)

Python 3.13 support

We added support for Python 3.13 (#3455). However, due to the Python object reference issue in PyTorch 2.7, we disabled the refitting related features for Python 3.13 in this release. This issue should be fixed in the next release.

What's Changed

Read more

Contributors

jingsh, narendasan, and 12 other contributors
Loading
0 Join discussion