index.rst

Torch-TensorRT

In-framework compilation of PyTorch inference code for NVIDIA GPUs

Torch-TensorRT is a inference compiler for PyTorch, targeting NVIDIA GPUs via NVIDIA's TensorRT Deep Learning Optimizer and Runtime. It supports both just-in-time (JIT) compilation workflows via the torch.compile interface as well as ahead-of-time (AOT) workflows. Torch-TensorRT integrates seamlessly into the PyTorch ecosystem supporting hybrid execution of optimized TensorRT code with standard PyTorch code.

More Information / System Architecture:

• Torch-TensorRT 2.0

Getting Started

• :ref:`installation`

.. toctree::
   :caption: Getting Started
   :maxdepth: 1
   :hidden:

   getting_started/installation
   getting_started/getting_started_with_windows

Dynamo Frontend

• :ref:`torch_compile`
• :ref:`dynamo_export`

.. toctree::
   :caption: Dynamo Frontend
   :maxdepth: 1
   :hidden:

   dynamo/torch_compile
   dynamo/dynamo_export

TorchScript Frontend

• :ref:`creating_a_ts_mod`
• :ref:`getting_started_with_python_api`
• :ref:`getting_started_cpp`
• :ref:`use_from_pytorch`

.. toctree::
   :caption: TorchScript Frontend
   :maxdepth: 1
   :hidden:

   ts/creating_torchscript_module_in_python
   ts/getting_started_with_python_api
   ts/getting_started_with_cpp_api
   ts/use_from_pytorch

FX Frontend

• :ref:`getting_started_with_fx`

.. toctree::
   :caption: FX Frontend
   :maxdepth: 1
   :hidden:

   fx/getting_started_with_fx_path

User Guide

• :ref:`dynamic_shapes`
• :ref:`ptq`
• :ref:`saving_models`
• :ref:`runtime`
• :ref:`using_dla`

.. toctree::
   :caption: User Guide
   :maxdepth: 1
   :hidden:


   user_guide/dynamic_shapes
   user_guide/ptq
   user_guide/saving_models
   user_guide/runtime
   user_guide/using_dla

Tutorials

• :ref:`torch_tensorrt_tutorials`
• :ref:`serving_torch_tensorrt_with_triton`
• :ref:`notebooks`

.. toctree::
   :caption: Tutorials
   :maxdepth: 3
   :hidden:

   tutorials/serving_torch_tensorrt_with_triton
   tutorials/notebooks
   tutorials/_rendered_examples/dynamo/torch_compile_resnet_example
   tutorials/_rendered_examples/dynamo/torch_compile_transformers_example
   tutorials/_rendered_examples/dynamo/torch_compile_advanced_usage
   tutorials/_rendered_examples/dynamo/torch_compile_stable_diffusion
   tutorials/_rendered_examples/dynamo/custom_kernel_plugins
   tutorials/_rendered_examples/distributed_inference/data_parallel_gpt2
   tutorials/_rendered_examples/distributed_inference/data_parallel_stable_diffusion

Python API Documenation

• :ref:`torch_tensorrt_py`
• :ref:`torch_tensorrt_logging_py`
• :ref:`torch_tensorrt_ptq_py`
• :ref:`torch_tensorrt_dynamo_py`
• :ref:`torch_tensorrt_ts_py`
• :ref:`torch_tensorrt_fx_py`

.. toctree::
   :caption: Python API Documenation
   :maxdepth: 0
   :hidden:

   py_api/torch_tensorrt
   py_api/logging
   py_api/ptq
   py_api/dynamo
   py_api/ts
   py_api/fx

C++ API Documenation

• :ref:`namespace_torch_tensorrt`
• :ref:`namespace_torch_tensorrt__logging`
• :ref:`namespace_torch_tensorrt__ptq`
• :ref:`namespace_torch_tensorrt__torchscript`

.. toctree::
   :caption: C++ API Documenation
   :maxdepth: 1
   :hidden:

   _cpp_api/torch_tensort_cpp
   _cpp_api/namespace_torch_tensorrt
   _cpp_api/namespace_torch_tensorrt__logging
   _cpp_api/namespace_torch_tensorrt__torchscript
   _cpp_api/namespace_torch_tensorrt__ptq

CLI Documentation

• :ref:`torchtrtc`

.. toctree::
   :caption: CLI Documenation
   :maxdepth: 0
   :hidden:

   cli/torchtrtc

Contributor Documentation

• :ref:`system_overview`
• :ref:`dynamo_converters`
• :ref:`writing_dynamo_aten_lowering_passes`
• :ref:`ts_converters`
• :ref:`useful_links`

.. toctree::
   :caption: Contributor Documentation
   :maxdepth: 1
   :hidden:

   contributors/system_overview
   contributors/dynamo_converters
   contributors/writing_dynamo_aten_lowering_passes
   contributors/ts_converters
   contributors/useful_links

Indices

• :ref:`supported_ops`
• :ref:`genindex`
• :ref:`search`

.. toctree::
   :caption: Indices
   :maxdepth: 1
   :hidden:

   indices/supported_ops

Legacy Further Information (TorchScript)

• Introductory Blog Post
• GTC 2020 Talk
• GTC 2020 Fall Talk
• GTC 2021 Talk
• GTC 2021 Fall Talk
• PyTorch Ecosystem Day 2021
• PyTorch Developer Conference 2021
• PyTorch Developer Conference 2022