Small LLMs on Jetson Orin Nano vs Orin NX: Which Model Fits 8 GB and 16 GB (2026)
Last updated: November 2026
The models people most often check on an Orin board are Llama 3.2 3B, Qwen2.5 7B, Phi-4-mini and Gemma 3 4B. This guide gives the memory each one needs on Orin Nano, Orin NX 8GB and Orin NX 16GB, how much context and concurrency each module leaves, and the generation speeds NVIDIA has published.
Quick Answer
- Orin Nano 8GB and Orin NX 8GB fit the same models. Memory, not compute, decides what loads, and both have 8 GB.
- On 8 GB, 3B–4B models at Q4 are the comfortable choice. Llama 3.2 3B needs about 4.1 GB, Phi-4-mini about 4.7 GB and Gemma 3 4B about 5.7 GB in total.
- 7B–8B models fit 8 GB only at Q4, with almost no margin. Qwen2.5 7B is at 84% of memory and Llama 3.1 8B at 91% (tight).
- Orin NX 16GB runs 7B–8B models at INT8, 3B–4B models at FP16, and many concurrent sequences or much longer contexts at Q4.
Who This Page Is For
- Builders of a local assistant, voice agent or robot brain deciding between Orin Nano and Orin NX.
- Anyone who already owns an 8 GB board and wants to know which model to run on it.
- Teams planning concurrency, such as several users or agents sharing one module, who need the sequence limit rather than a yes/no fit.
Memory Decides, Not Compute
A language model has to hold its weights and its KV cache in memory for as long as it serves. On Jetson that memory is one unified pool shared by the CPU, the GPU and the operating system. A model either fits in what is left, or it swaps and stalls.
Orin Nano and Orin NX 8GB both have 8 GB of that pool, so every figure below is identical on the two. The NX adds compute, which can raise generation speed, but it does not let you load a larger model or a longer context. Orin NX 16GB doubles the pool, and that is the change that matters for model size.
Memory Fit Table: 8 GB vs 16 GB
Total memory in GB at a 4,096-token context, one sequence, llama.cpp, a headless system and a 10% reserve. The last two columns give the verdict on an 8 GB module (Orin Nano or Orin NX 8GB) and on Orin NX 16GB. Model names link to every module and quantisation on the model-fit pages.
| Model | Parameters | Q4 | INT8 | FP16 | 8 GB module | Orin NX 16GB |
|---|---|---|---|---|---|---|
| Llama 3.2 1B | 1.24 B | 2.4 | 3.0 | 4.2 | All fit | All fit |
| Llama 3.2 3B | 3.21 B | 4.1 | 5.6 | 8.9 | Q4, INT8 fit; FP16 does not | All fit |
| Phi-4-mini | 3.8 B | 4.7 | 6.3 | 10.2 | Q4, INT8 fit; FP16 does not | All fit |
| Gemma 3 4B | 4.3 B | 5.7 | 7.4 | 11.4 | Q4 fits; INT8 tight; FP16 does not | All fit |
| Qwen2.5 7B | 7.61 B | 6.7 | 10.4 | 18.2 | Q4 only (84%) | Q4, INT8 fit; FP16 does not |
| Mistral 7B v0.3 | 7.25 B | 6.7 | 10.3 | 17.7 | Q4 only (84%) | Q4, INT8 fit; FP16 does not |
| Llama 3.1 8B | 8.03 B | 7.3 | 11.2 | 19.4 | Q4 tight (91%) | Q4, INT8 fit; FP16 does not |
Prefilled runs to start from:
Context and Concurrency Headroom
"It loads" is the first question. The second is how much room is left for longer prompts or more than one conversation at a time. The KV cache grows with both. The figures below are at Q4 and use the calculator's planning limit: the largest value that stays under 85% of memory.
| Model (Q4) | Concurrent sequences at 4,096 tokens: 8 GB | Same: Orin NX 16GB | Context at one sequence: 8 GB | Same: Orin NX 16GB |
|---|---|---|---|---|
| Llama 3.2 3B | 6 | 20 | 16K | 64K |
| Phi-4-mini | 4 | 17 | 16K | 64K |
| Gemma 3 4B | 2 | 13 | 8K | 32K |
| Qwen2.5 7B | 1 | 29 | 4K | 32K (the model's maximum) |
| Llama 3.1 8B | 0 (tight even at one) | 12 | none (tight even at 1K) | 32K |
This is where 16 GB earns its price. On 8 GB a 7B model serves one short conversation. On 16 GB the same model serves dozens of 4K conversations, or one conversation with the model's full 32K context.
Published Generation Speeds
NVIDIA's Jetson AI Lab has published generation rates for several of these models on Orin Nano. These are the figures we could trace to a published source:
| Model | Module | Tokens per second | Source |
|---|---|---|---|
| Llama 3.2 3B | Orin Nano Super | 43 | Jetson AI Lab benchmarks |
| Qwen2.5 7B | Orin Nano Super | 22 | Jetson AI Lab benchmarks |
| Llama 3.1 8B | Orin Nano Super | 19 | Jetson AI Lab benchmarks |
| Llama 3.2 3B | Orin Nano (MLC, 4-bit) | 27.7 | Jetson AI Lab small-LLM tutorial |
| Llama 3.2 1B | Orin Nano (MLC, 4-bit) | 54.8 | Jetson AI Lab small-LLM tutorial |
Two cautions. First, the Super and non-Super figures come from different pages, so they are not a clean comparison of the two power modes. Second, we have no published Orin NX figures on record for these models. The NX's extra compute should help generation speed, but measure it rather than assume it.
Quantisations Orin Runs
- 4-bit weight formats, such as GGUF Q4 on llama.cpp or MLC's 4-bit builds, are what make 7B–8B models possible on 8 GB.
- INT8 runs on every Orin module and is the usual quality step up on 16 GB.
- FP16 is the unquantised reference. On these modules it is practical only for models up to about 4B on 16 GB.
- FP8 and NVFP4 do not run on Orin. NVIDIA's TensorRT support matrix lists FP8 and FP4 as not available on compute capability 8.7, which is Jetson Orin. Those formats are Jetson Thor features.
Which Module to Buy
- Orin Nano 8GB (or Super): one user, a 1B–4B model, prompts up to a few thousand tokens.
- Orin NX 8GB: the same model sizes as Orin Nano, with more compute. Choose it for speed or for the rest of your workload, not to fit a larger model.
- Orin NX 16GB: 7B–8B models at INT8, long contexts, several concurrent users, or an LLM beside a camera pipeline on the same module.
- Beyond 16 GB: for 13B and larger models, or many agents at once, see the model-fit matrix for AGX Orin and Thor.
Frequently Asked Questions
Does Orin NX 8GB run bigger LLMs than Orin Nano 8GB?
No. Both have 8 GB of unified memory, so the same models fit at the same quantisations, with the same context and concurrency limits. The Orin NX 8GB adds compute, which can raise generation speed, but it does not raise the size of model you can load. For larger models or longer contexts you need the Orin NX 16GB.
Can Qwen2.5 7B or Llama 3.1 8B run on an Orin Nano?
Only at Q4, and only just. At a 4,096-token context and one sequence, the Model Memory Fit engine puts Qwen2.5 7B at about 6.7 GB in total on an 8 GB module (84% of memory, just under the 85% line where it calls a result tight). Llama 3.1 8B comes to about 7.3 GB (91%, tight). INT8 and FP16 do not fit either model. On an Orin NX 16GB both run at INT8 with room for many concurrent sequences.
What is the best LLM for an 8 GB Jetson?
For headroom, a 3B to 4B model at Q4: Llama 3.2 3B (about 4.1 GB), Phi-4-mini (about 4.7 GB) or Gemma 3 4B (about 5.7 GB). Each leaves memory for a longer context or a second sequence. Llama 3.2 3B and Phi-4-mini also fit at INT8. 7B and 8B models load at Q4 but leave almost nothing for anything else on the device.
Can I use FP8 or NVFP4 on Jetson Orin to save memory?
No. NVIDIA's TensorRT support matrix lists FP8 and FP4 as not available on compute capability 8.7, which is Jetson Orin. On Orin, the memory-saving options are INT8 and 4-bit weight formats such as GGUF Q4 or MLC's 4-bit builds. FP8 and FP4 are Jetson Thor features.
Bottom line
On Jetson Orin, memory decides which LLM you can run, and Orin Nano and Orin NX 8GB have the same 8 GB. Both run 3B–4B models comfortably at Q4, and 7B–8B models only at Q4 with no room to spare. Orin NX 16GB runs 7B–8B models at INT8, with long contexts or many concurrent users. Pick the module by the model size and concurrency you need, then confirm speed on the hardware.
Size your model on your module, at your context length and concurrency.
Open Model Memory Fit