Multi-Model Inference on Jetson (2026): Running Detection, Classification & Pose Concurrently
Updated July 2026
Real edge pipelines rarely run one model. You detect, then classify the crop; you track, then read a plate; you find a person, then estimate pose. Running several models on one Jetson is a memory problem long before it's a compute problem — here's how to size it, where the DLA helps, and which board actually fits.
Quick Answer
- The constraint is unified memory, not TOPS. Every concurrent model keeps its weights, engine, and activation buffers resident at once — plus a CUDA/TensorRT context. You run out of RAM before you run out of compute.
- Orin NX 16GB is the mainstream pick. For a detector plus one or two secondary models on 2–4 cameras, the 16GB of memory is what matters — and it's what the Hardware Selector lands on for that shape of workload.
- AGX Orin 32/64GB for heavy pipelines. Eight-plus cameras, three-plus concurrent models, or large VLM/generative stages — and its two DLA engines let you run secondary models off the GPU.
- Orin Nano Super (8GB) is a one-secondary-model board. A detector plus one small classifier is realistic; a third model usually won't fit. It has no DLA, so everything shares the GPU.
- Use DeepStream for camera pipelines, Triton for serving. Both handle batching and model concurrency so you're not hand-scheduling CUDA streams.
Who This Page Is For
- Building a perception pipeline with more than one model (detect → classify, track → read, detect → pose)
- Deciding whether a workload fits an Orin Nano, needs an Orin NX 16GB, or an AGX Orin
- Hitting out-of-memory errors or latency cliffs when adding a second or third model
- Choosing between DeepStream, Triton, and raw TensorRT for multi-model orchestration
What "Multi-Model" Actually Means
"Multi-model" isn't a bigger model — it's several models running on one device at the same time, usually chained so the output of one feeds the next. The common shapes at the edge:
- Detect → classify — a detector finds objects, a lightweight classifier labels each crop (vehicle make/model, product SKU, defect type).
- Detect → track → read — detection plus a tracker plus OCR for licence plates or container codes.
- Detect → pose / attributes — person detection feeding a pose estimator or an attribute model (PPE, fall detection, re-identification).
- Vision + language — a detector plus a small VLM or captioner for scene description or anomaly narration.
The reason this deserves its own sizing conversation is that a multi-model pipeline behaves nothing like a single model scaled up. Two 12GB-class models are not "one 24GB model" — they contend for memory, for the GPU, and for the latency budget in ways that a single benchmark FPS number never captures. Get the sizing wrong and you don't get a slow pipeline; you get an out-of-memory crash or a silent fallback that quietly costs you an order of magnitude.
The Three Real Constraints
1. Unified memory — the binding one
Jetson modules use unified memory: one LPDDR5 pool shared by the CPU and GPU. There is no separate VRAM. That pool holds your OS, your application, camera and encode/decode buffers, and — for every model you load concurrently — its weights, its TensorRT engine, and its activation workspace. Each model also brings a CUDA/TensorRT runtime context that can cost several hundred megabytes before a single inference runs. Load three models across four streams on an 8GB Orin Nano and you can exhaust memory before compute is ever the issue. This is why memory tier, not TOPS, is the first thing to size.
2. The GPU time-slice — models share, not multiply
The integrated GPU runs one kernel at a time. Running several models "concurrently" via CUDA streams interleaves them — it does not run them in true parallel on the GPU. So the honest math is that your models share the GPU's throughput: a detector that hits 120 FPS alone and a classifier that hits 300 FPS alone will each get less when they run together, roughly in proportion to how much GPU time each consumes. Plan your frame budget against the sum of per-model compute, not the fastest one.
3. The DLA — where real parallelism comes from
Orin NX and AGX Orin include dedicated DLA (Deep Learning Accelerator) engines — two on AGX Orin — that run INT8/FP16 models independently of the GPU. Placing a secondary model (a classifier or a small pose net) on a DLA means it runs while the GPU handles the primary detector: genuine overlap, and it frees GPU cycles for the model that needs them most. The Orin Nano has no DLA, so on it every model competes for the same GPU — a real, often-overlooked reason the Nano struggles with multi-model where the NX and AGX shine.
Memory Math: Sizing the Real Ceiling
A back-of-the-envelope estimate that will keep you off the memory cliff:
| Model engines | Sum of each model's TensorRT engine size (~ its INT8/FP16 weight footprint). A YOLOv8s INT8 engine is tens of MB; a ResNet-class classifier similar; a small VLM can be several GB. |
| Activations | Per-model working memory × concurrent streams. Grows with resolution, batch size, and stream count — often the sneaky term. |
| Runtime context | CUDA + TensorRT overhead, roughly 0.8–1.5GB, largely fixed regardless of model size. Multiple processes multiply it. |
| OS + application | Reserve 2–3GB for JetPack, your pipeline code, camera buffers, and encode/decode. |
| Headroom | Stay a couple of GB below the ceiling. Being memory-tight triggers swapping that destroys latency long before you see an OOM. |
The practical takeaway: the fixed costs (runtime context + OS) already claim 3–4GB before your first model loads. That's why an 8GB board is a one-or-two-model device, a 16GB board comfortably holds three-plus model engines across several streams, and 32–64GB is what large VLM or generative stages need. Size it before you buy with the Memory Estimator, and pin the platform with the Hardware Selector.
Which Jetson Runs Which Pipeline
The recommendations below track what the Hardware Selector returns for multi-model workloads — sized on the memory reality above, not headline TOPS.
| Board | Memory / DLA | Realistic multi-model workload |
| Orin Nano Super 8GB — ~$249 | 8GB · no DLA | A detector plus one small secondary model on 1–2 streams. The entry point — realistic, but a third model rarely fits, and everything shares the GPU. |
| Orin NX 16GB — ~$599 | 16GB · DLA | The mainstream multi-model workhorse. Detector + one or two secondaries across 2–4 cameras. 16GB holds several engines resident; DLA offloads a secondary. The Selector's default pick for this shape. |
| AGX Orin 32GB — ~$899 | 32GB · 2× DLA | Eight-plus cameras or three-plus concurrent models. 200 INT8 TOPS and two DLA engines give real headroom for demanding pipelines. |
| AGX Orin 64GB — ~$1,599 | 64GB · 2× DLA | Multi-model plus a sizeable VLM/generative stage, or very high stream counts where the 64GB memory is the whole point. |
| Jetson Thor T5000 — ~$3,499 | 128GB · next-gen | Generative-heavy multi-model, large VLMs, or 20+ camera density — where you're combining perception with an on-device language model. |
Note the pattern: as pipelines get heavier the deciding factor is memory (8 → 16 → 32 → 64 → 128GB), with DLA availability the second axis. TOPS matters, but it's rarely the number that stops you.
How to Actually Run It
DeepStream — the default for camera pipelines
NVIDIA DeepStream is purpose-built for exactly this: a primary nvinfer detector feeding secondary nvinfer classifiers, with multi-stream batching (nvstreammux) and object tracking (nvtracker) handled for you. Secondary models run only on the objects the primary detected, which is far cheaper than running every model on every full frame. If your workload is "cameras in, structured events out," start here.
Triton Inference Server — serving several models
When your models are consumed as a service (request in, prediction out) rather than a fixed camera pipeline, Triton has a Jetson build that serves multiple models with dynamic batching and configurable per-model concurrency. It's the right tool when different parts of your application call different models on demand.
Raw TensorRT + CUDA streams — full control
For custom scheduling, build one TensorRT engine per model and drive them with separate CUDA streams, explicitly assigning INT8 secondaries to the DLA (--useDLACore) so they overlap the GPU's work. More code, but maximum control over placement and latency — worth it when you're squeezing a tight frame budget.
Three Mistakes We See Most Often
1. Sizing on TOPS instead of memory
The single most common error. A board can have plenty of TOPS for your combined models and still be unable to hold them all in memory at once. Add up engine sizes, activations, runtime context, and OS reserve first — then check that compute is sufficient. Memory is almost always the wall you hit first.
2. Assuming models run in parallel on the GPU
Loading three models and expecting three times the work is a recipe for a blown latency budget. The GPU time-slices; your frame budget has to cover the sum of per-model GPU time. If that sum exceeds your budget, move a model to the DLA or drop to a lighter secondary — don't just hope concurrency hides it.
3. Running the Nano to the edge of its memory
An 8GB Orin Nano that fits your pipeline "with 300MB to spare" will swap under real camera load, and swapping on unified memory is catastrophic for latency. If you're within a couple of gigabytes of the ceiling, that's the signal to step up to the 16GB Orin NX — the memory headroom, not the extra TOPS, is what you're buying.
Frequently Asked Questions
Which Jetson is best for running multiple models at once?
For most multi-model camera pipelines — a detector plus one or two secondary models on 2–4 streams — the Orin NX 16GB is the pragmatic pick, because its 16GB of unified memory holds several engines resident at once. Step up to AGX Orin 32GB or 64GB for 8+ cameras or three-plus concurrent models, where the two DLA engines let you offload secondaries off the GPU. The 8GB Orin Nano Super only comfortably runs a detector plus one small secondary; beyond that it runs out of memory before it runs out of compute.
Why does multi-model inference need more RAM than a single model?
Every concurrently-loaded model keeps its own weights, TensorRT engine, and per-stream activation buffers resident at the same time — and each also carries a CUDA/TensorRT runtime context that can cost several hundred megabytes on its own. On Jetson that all comes from one shared LPDDR pool (unified memory) also feeding the OS, camera buffers, and your application. So multi-model deployments are almost always memory-bound before they are compute-bound.
Can Jetson run multiple models truly in parallel?
The integrated GPU executes one kernel at a time and time-slices between models via CUDA streams, so several GPU models share — not multiply — total throughput. True parallelism comes from the DLA engines on Orin NX and AGX Orin: you can run a secondary INT8 model on a DLA while the GPU runs the primary detector, genuinely overlapping the two. The Orin Nano has no DLA, so on it every model competes for the same GPU.
What framework should I use for multi-model pipelines on Jetson?
NVIDIA DeepStream is the canonical choice for camera pipelines: a primary nvinfer detector feeding secondary classifiers, with batching and tracking handled across streams. For request/response serving of several models with dynamic batching, use Triton Inference Server (it has a Jetson build). For full control, build a TensorRT engine per model and drive them with CUDA streams, placing INT8 secondaries on the DLA.
How do I estimate whether my models fit on a given Jetson?
Sum each model's engine size (roughly its INT8/FP16 weight footprint), add per-stream activation memory times your stream count, add a CUDA/TensorRT runtime overhead of roughly 0.8–1.5GB, then reserve 2–3GB for the OS and application. Compare that total against the board's unified memory. If you're within a couple of gigabytes of the ceiling, move a secondary model to the DLA or step up a memory tier. Our Memory Estimator does this math for you.
Bottom line
Multi-model is a memory problem wearing a compute costume. Size unified memory first — engines + activations + runtime context + OS reserve — then confirm the GPU (and DLA) can cover the summed compute inside your frame budget. In practice that lands most camera pipelines on the Orin NX 16GB, pushes heavy or generative pipelines to AGX Orin, and leaves the 8GB Nano for one-secondary-model workloads.
Size your multi-model pipeline
Start with the Hardware Selector to pin the board on memory, then run the Memory Estimator to confirm your specific models fit under the unified-memory ceiling.