What is Torch-TensorRT?

PyTorch ecosystem

Script mode

Compiling with Torch-TensorRT

Conclusion

What is TensorFlow-TensorRT?

How does TensorRT work?

Benefits of single-line integration

Installation

Intro

How to convert a model to TensorRT

Outro

Intro

What is TensorRT?

What do you need to run TensorRT?

TensorRT conversion script

Converting the model

Analyzing the graph

Tensorboard graph

Restart with log directory

Static values in models

Conclusion

Intro

Tensor Fusion

When not to use TensorRT?

Setup steps

Optimizing Inference

TensorRT Batch Size

Running the Model

Running PyTorch and TensorRT

Question: When you convert models for inference, do you see any performance losses as a result?

Conclusion

What is TensorFlow-TensorRT?

Getting started

Image Classification Example

Pre-trained Model Predictions

Export TF-Save Model

Optimizing the Model

TensorRT Engines

Building TensorRT Engines

Half-Precision

Conclusion

Introduction

Starting Jupyter

Setting Up TensorRT-LLM

BLOOM

Converting the model from HuggingFace

Building the default-optimized model

Building the quantized model (INT8)

Quantization

How does TensorRT-LLM work (high-level)?

Benchmarking

Visualizing the Results

Conclusion & Where To Go

Requirements

Check the Version on Ubuntu

Docker

Building the Tensorrt Docker Image

Build the Tensorrt Docker Image

Compile and Install the Tensor

Demo

Prerequisites

Convert to Tensor Rt Engine

Benchmark

intro

clone Torch-TensorRT

install and setup Docker

install Nvidia Container Toolkit & Nvidia Docker 2

Torch-TensorRT container (option #1)

Torch-TensorRT Nvidia NGC container (option #2)

import Pytorch

load ResNet50

load sample image

sample image transforms

batch size

prediction with ResNet50

softmax function

ImageNet class number to name mapping

predict top 5 classes of sample image (topk)

speed test benchmark function

CPU benchmarks

CUDA benchmarks

trace model

convert traced model into a Torch-TensorRT model

TensorRT benchmarks

download Jupyter Notebook

HOW DID I MISS THIS???

thanks for watching!

初めに

インストール

SD1.5での速度、画質、VRAMの比較

SDXLの比較

Get Started with Tensor Rt

Import the Tensorflow Model into Tensor

Add an Additional Top K Layer

Run the Engine

Inference

Intro

What is TensorRT?

TensorRT goals

How did TensorRT get started?

How do I get started?

What's the performance like on a V100?

How does TensorRT work?

How does TensorRT streamline the data flow?

Where can developers learn about TensorRT?

Conclusion

Overview of the Project Structure

Configuration Options

Build Function

Check Cuda Error Code

Preparation of the Inputs

Size Keep Aspect Pre-Show

Run Inference Method

Error Checking

Normalization

Copy the Outputs

Intro

What is TensorRT?

Vertical kernel fusion

TensorRT Calibration

TensorRT Converters

Saving a PyTorch model

Compiling TensorRT

Image Classification Example

Inference performance plots

Q&A session

Intro

Why TensorRT?

Why is there no libtensorrt?

Offline steps

API Overview

Regenerating the Engine

Switching optimization profiles

Running Inference

Image formats

Conclusion

Intro

Applications of edge computing

What is TensorRT?

Freezing graphs

Run the conversion script

Image acquisition

Can the conversion script be run on the Jetson?

Python support for TensorRT on Jetson?

Will TensorFlow be provided in Jetson?

Conclusion