YOLO on Jetson with TensorRT: FP16 vs INT8 Benchmarks & Export Gotchas (2026)

Quick Answer

Export your YOLO model to a TensorRT engine and run FP16 by default — it roughly doubles throughput over FP32 with negligible accuracy loss. Move to INT8 only when camera count or power forces it, and validate the mAP drop on your own footage first. TensorRT delivers 2–3× the throughput of native PyTorch by fusing layers and auto-tuning kernels for your specific GPU. The two traps that burn people: on JetPack 6, TensorRT 10.3 blocks INT8 end-to-end builds (upgrade to 10.7+), and the DLA improves efficiency, not single-stream latency. Always rebuild the engine on the exact target module — engines aren't portable across hardware or TensorRT versions.

Who This Page Is For

• Engineers deploying YOLO (v8, v11, or newer) on Jetson who are still running PyTorch and wondering why FPS is low.
• Anyone choosing a precision and needing to know what INT8 actually costs in accuracy before committing.
• Builders sizing camera count per module who need real TensorRT FPS, not theoretical TOPS.
• People stuck on a failing INT8 export on JetPack 6 who want the fix, not another forum thread.

How to Use This Page

1. Confirm you're on a GPU export path. TensorRT or TorchScript use the GPU; TFLite, NCNN, and plain ONNX runtime fall back to CPU and are far slower.
2. Pick a precision from the trade-off table. Start at FP16; consider INT8 only if you're throughput- or power-bound.
3. Read the gotchas before you export. The TensorRT 10.3 INT8 issue and DLA fallback behavior will save you hours.
4. Size camera count from real FPS. Use the benchmark figures, then apply your detect-fps target.
5. Validate with tools. Run your model and stream count through GPU Sizing and the System Designer to confirm headroom before buying hardware.

Why TensorRT At All

TensorRT is NVIDIA's inference compiler. It takes your trained model — exported to ONNX — and rebuilds it into a .engine optimized for one specific GPU through three mechanisms: layer fusion (collapsing consecutive operations into single kernels), precision calibration (running in FP16 or INT8 instead of FP32), and kernel auto-tuning (picking the fastest implementation for your hardware). The net effect on Jetson is commonly 2–3× the throughput of the same model in native PyTorch, and a fraction of the per-frame latency.

This is why a YOLO model that feels sluggish in Python is not a hardware problem — it's a deployment-format problem. The Jetson's GPU is idling while PyTorch overhead and the Python interpreter throttle the pipeline. The fix is to get off the PyTorch inference path and onto a compiled TensorRT engine. The only real questions are which precision to compile for and how to dodge the export traps.

FP32 vs FP16 vs INT8: The Trade-off

The headline numbers: on AGX Orin, YOLOv8s through trtexec runs around 139 FPS at FP32 and around 313 FPS at INT8 — more than double. FP16 sits comfortably in between with effectively no accuracy cost, which is why it's the default recommendation. INT8 is the fastest path and the only one that needs calibration: you feed TensorRT ~1000 representative images so it can learn the activation ranges to quantize against. Skip good calibration and the accuracy drop is worse than it needs to be. The mantra: FP16 unless you're forced to INT8, and prove the INT8 accuracy on your own data.

Real FPS Benchmarks

Representative TensorRT figures from vendor and community benchmarks (single stream, 640×640 input, YOLOv8-class models). Treat as planning approximations — exact numbers vary with model variant, JetPack/TensorRT version, and power mode.

Power for lightweight YOLO under TensorRT on Orin Nano / Orin NX typically lands in the 7–14W range. The takeaway isn't the exact FPS — it's the shape: even the smallest Orin runs a nano-class YOLO comfortably in real time once it's on TensorRT, and the bigger modules have enormous headroom at INT8. The bottleneck in a real deployment is usually decode and stream count, not single-model inference.

The Export Workflow

The path is short once you know it:

1. Flash JetPack (JetPack 7.2 is current as of mid-2026; much of the Orin YOLO toolchain still targets the legacy 6.x line) so CUDA, cuDNN, and TensorRT — including the trtexec tool — are present.
2. Use the prebuilt JetPack 6 Docker container for your YOLO toolchain rather than fighting ARM64 pip wheels — the pip-installed PyTorch/Torchvision are not compatible with aarch64 Jetson, so the container saves the manual wheel dance.
3. Export PyTorch → ONNX → TensorRT engine, choosing FP16 or INT8 at export time. For INT8, point the exporter at your calibration image folder and batch size so it builds a calibration table.
4. Benchmark with trtexec to get a clean mean-latency number (then FPS = 1000 / latency_ms), independent of any display sink cap.
5. Run inference from the resulting .engine. Enable max-performance power mode for benchmarking.

For multi-stream production rather than a single feed, pair the engine with DeepStream — see the DeepStream vs Frigate guide linked below for when to reach for it.

The Gotchas That Waste an Afternoon

• The TensorRT 10.3 INT8 bug (JetPack 6). TensorRT 10.3, which ships with JetPack 6, has a known issue that blocks INT8 engine builds when the end-to-end (NMS-included) branch is enabled. Some toolchains silently disable the end2end branch so the export at least succeeds; to keep end2end INT8, upgrade TensorRT to 10.7+ via the CUDA apt repo and re-run. This is the single most common "my INT8 export fails" cause on JP6.
• Engines aren't portable. A .engine is compiled for a specific GPU architecture and TensorRT version. Build it on an AGX Orin and it won't load on an Orin Nano; build it under one JetPack and it may fail under another. Rebuild on the target, or script the build at first boot.
• CPU-only export formats look fine and run slow. TFLite, NCNN, MNN, and plain ONNX runtime export without error but run on the CPU. If your FPS is mysteriously low, confirm you're actually on TensorRT or TorchScript, not a CPU path.
• Newer YOLO architectures can break the TensorRT graph. Some experimental YOLO releases export with post-processing subgraphs that don't convert cleanly, producing bounding-box drift or bad confidence scores in the engine. Validate detections after export; don't assume a clean export means correct output.
• Display-capped FPS. If you benchmark through a live display sink, you may read the monitor's refresh rate (e.g. 60 FPS) instead of true throughput. Use trtexec or a headless sink for real numbers.

When to Use the DLA

Orin modules include Deep Learning Accelerator (DLA) cores alongside the GPU, and YOLO can export to run on a DLA core (FP16 or INT8 only). The common misconception is that the DLA is "faster." It isn't — its purpose is throughput per watt and GPU offload, not lower latency. In fact, some layers aren't DLA-supported and fall back to the GPU, which can add latency. So the DLA is the right tool when you want to cut power draw or free the GPU to run a second model or other pipeline work concurrently — not when you're chasing the lowest single-stream latency. For a power-constrained battery or sealed deployment, routing detection to the DLA can be a meaningful efficiency win; for a plugged-in box that just needs maximum FPS, keep it on the GPU.

Turning FPS Into Camera Count

Single-model FPS is only step one. For a multi-camera deployment the useful number is how many streams the module sustains, which depends on your detect-fps target and whether you run one shared model or one model per stream. As a reference point, an Orin NX 16GB running a single shared YOLOv8s INT8 model can feed on the order of dozens of cameras at a low detect rate, but switching to a separate model per stream drops that sharply because each model instance consumes RAM. A bare Orin Nano 8GB realistically supports a handful of streams; an Orin NX 16GB roughly triples that.

And remember the perennial caveat: detection is not decode. Your camera-count ceiling is often set by how many H.264/H.265 streams the host can hardware-decode, not by inference. Size both. The GPU Sizing and System Designer tools turn your model, precision, and stream count into a concrete capacity check.

Decision Framework

Use FP16 if:

• You want the production default with no meaningful accuracy cost
• Your camera count and power budget aren't pushing the module to its limit
• You'd rather not maintain a calibration dataset

Use INT8 if:

• You're throughput-bound (high camera count) or power-bound (battery, sealed enclosure)
• You can assemble ~1000 representative calibration images and validate the mAP drop
• The extra ~1.5–2× over FP16 changes whether the deployment fits

Route to the DLA if:

• You need to lower power draw or offload the GPU for a second workload
• You can tolerate slightly higher latency from GPU fallback on unsupported layers
• Efficiency, not peak single-stream FPS, is the goal

Stop and check if:

• Your INT8 export fails on JP6 — upgrade TensorRT past 10.3
• FPS is suspiciously low — confirm you're on a GPU export path, not CPU
• Detections look wrong after export — validate the graph converted cleanly

Frequently Asked Questions

The Bottom Line

If your YOLO model is running in PyTorch on a Jetson, you're using a fraction of the board. Export to a TensorRT engine, run FP16 by default, and reach for INT8 only when camera count or power demands it — with real calibration and an accuracy check on your own footage. Dodge the two big traps (the TensorRT 10.3 INT8 issue on JP6, and the DLA-is-efficiency-not-speed misconception), never ship an engine across different hardware, and remember that decode, not inference, usually sets your camera ceiling.

Size your model, precision, and stream count against the target module with the tools below before you commit hardware.

Recommended Reading

• DeepStream vs Frigate (2026) — Which multi-stream analytics stack to wrap around your TensorRT engine.
• Best Hardware for Frigate NVR (2026) — Detector selection if you'd rather use a turnkey NVR than build a pipeline.
• Jetson Orin Nano vs Orin NX (2026) — Choosing the module for your camera count and model size.
• Jetson Orin Nano Power Modes — How power mode interacts with TensorRT throughput.
• GPU Sizing Tool — Estimate inference throughput for your model and precision on a target Jetson.
• System Designer — Turn model, precision, and stream count into a capacity check and BOM.
• Hardware Selector — Filter platforms by task, power, and memory with ranked alternatives.