Speech Model Sizing

Whisper on Jetson: Which Speech Model Fits Orin Nano, Orin NX and Thor (2026)

Last updated: November 2026

Speech-to-text is one of the most common reasons people size a Jetson for a language model. This guide gives the memory each Whisper model needs on each module, at the quantisations Jetson Orin actually runs, and separates what has been measured from what has not.

small: 1.6 GB at Q4
medium: 3.0 GB at Q4
large-v3: 6.5 GB at Q4
All three fit 8 GB at Q4

Quick Answer

  • Whisper small and medium fit every Jetson module at Q4, INT8 and FP16: about 1.6–1.9 GB and 3.0–4.2 GB in total.
  • Whisper large-v3 on an 8 GB Orin Nano or Orin NX 8GB: fits at Q4 (about 6.5 GB, 81% of memory), is tight at INT8 (about 7.3 GB, 91%), and does not fit at FP16 (about 8.9 GB).
  • Orin NX 16GB and larger run large-v3 at FP16 with room to spare.
  • Speed: the only published Orin measurements are for tiny.en and base.en. NVIDIA reports TensorRT transcribing 20 seconds of speech in 0.64 s and 0.86 s on Orin Nano.

Who This Page Is For

  • Builders adding voice input or transcription to a robot, kiosk, intercom or meeting device, and choosing the Jetson to run it on.
  • Teams running speech beside a camera pipeline who need to know how much memory Whisper leaves for everything else.
  • Anyone deciding between Whisper sizes on a fixed 8 GB or 16 GB module.
Speech vs Text Models

Why Speech Models Size Differently

Whisper is an encoder–decoder model. The encoder turns about 30 seconds of audio into features, and the decoder writes the transcript one token at a time. Two things follow for memory:

  • The KV cache is small. The decoder's text limit is 448 tokens, so the cache that dominates a chat LLM's memory at long contexts stays modest here.
  • Weights × bytes understates the real footprint. Encoder activations, audio and beam-search buffers, and allocator slack add memory that a parameter count does not show. OpenAI's Whisper README publishes the memory each size needed at FP16: about 5 GB for medium and about 10 GB for large. For those two, the engine carries the gap between the published figure and its own modelled parts as fixed serving overhead, added at every quantisation. For small, the published figure (about 2 GB) is at or below the engine's own estimate, so the engine adds nothing.

That second point is why the gap between Q4 and FP16 is narrower than the weight sizes suggest. For large-v3 the weights shrink by about 4×, but the total drops from about 8.9 GB to about 6.5 GB.

Fit Table

Whisper Memory Fit Table

Total memory in GB at one sequence on a headless system, with a 10% reserve. 8 GB modules are Orin Nano (including Super) and Orin NX 8GB. 16 GB and larger covers Orin NX 16GB, AGX Orin 32GB and 64GB, and Thor T4000 and T5000; every cell in that column fits. Each figure links to its full breakdown on the model-fit pages.

Model Parameters Q4 INT8 FP16 On an 8 GB module
Whisper small 0.24 B 1.6 1.7 1.9 Fits at every quantisation
Whisper medium 0.77 B 3.0 3.4 4.2 Fits at every quantisation
Whisper large-v3 1.55 B 6.5 7.3 8.9 Q4 fits (81%), INT8 tight (91%), FP16 does not fit

Open a prefilled run to change the module, quantisation or reserve:

Quantisation

Which Quantisation to Use

  • FP16 is Whisper's published reference precision. Use it when memory allows: every size on a 16 GB module, and small or medium on 8 GB.
  • INT8 runs on every Orin module and saves about 1.6 GB on large-v3. On 8 GB modules large-v3 at INT8 is tight, so expect pressure if anything else is resident.
  • Q4, a 4-bit weight format such as GGUF Q4, is the only way large-v3 fits an 8 GB module with comfortable margin. Check the accuracy cost on your own audio before you commit.
  • FP8 and NVFP4 are not options on Orin. NVIDIA's TensorRT support matrix lists FP8 and FP4 as not available on compute capability 8.7, which is Jetson Orin. Jetson Thor supports them, but large-v3 already fits a Thor module at FP16.
Speed

What Speed Has Been Measured

Memory decides whether Whisper loads; speed decides whether it keeps up. The only published Whisper timings on Jetson Orin we could trace come from NVIDIA's whisper_trt project. It reports time to transcribe a 20-second clip on Jetson Orin Nano:

Model PyTorch Whisper faster-whisper whisper_trt (TensorRT)
tiny.en 1.74 s 0.85 s 0.64 s
base.en 2.55 s not reported 0.86 s

Both models finish 20 seconds of speech in under a second with TensorRT, far faster than real time. The README does not state the JetPack version or power mode.

No comparable published measurement covers small, medium or large-v3 on any Orin module. Larger Whisper models do more work per second of audio, so do not extrapolate the tiny and base figures. Measure on your target module, at the power mode you will deploy with, before you commit to large-v3 for live transcription.

Shared Memory

Running Whisper Beside Other Work

Jetson modules have one pool of unified memory shared by the CPU, the GPU, the operating system and every model on the device. The totals above already include a headless OS and a 10% reserve. Anything else you run, such as a camera pipeline, a chat model or a desktop session, comes out of what is left.

  • In Model Memory Fit, set the memory used by co-located workloads to see whether Whisper still fits beside them.
  • Choosing a desktop OS instead of headless adds operating-system memory; the calculator shows the difference.
  • If speech is one stage of a larger pipeline, size the whole device in Architecture Designer rather than each model on its own.

Frequently Asked Questions

Can Whisper large-v3 run on a Jetson Orin Nano 8GB?

Yes, quantised. At Q4 the Model Memory Fit engine puts Whisper large-v3 at about 6.5 GB in total on an 8 GB module — weights, decoder KV cache, runtime overhead, the operating system and a 10% reserve — which is about 81% of memory and fits. At INT8 it is about 7.3 GB, which is tight, and at FP16 about 8.9 GB, which does not fit. Any 16 GB or larger module runs it at every one of those quantisations.

Which Whisper model should I use on an 8 GB Jetson?

Whisper small (about 1.6–1.9 GB in total) and Whisper medium (about 3.0–4.2 GB) fit an 8 GB Orin Nano or Orin NX at Q4, INT8 or FP16 with room left for other work. Whisper large-v3 fits only at Q4 with comfortable margin; at INT8 it uses about 91% of memory. Choose by the accuracy you need and the memory the rest of the device needs, not by whether the model loads.

How fast is Whisper on Jetson Orin Nano?

NVIDIA's whisper_trt project reports, on Jetson Orin Nano, 0.64 seconds to transcribe a 20-second clip with tiny.en and 0.86 seconds with base.en using TensorRT, against 1.74 and 2.55 seconds for the reference PyTorch Whisper. No comparable published measurement covers small, medium or large-v3 on Orin, so treat speed for those as something to measure on your own hardware.

Can I run Whisper in FP8 or NVFP4 on Jetson Orin?

No. NVIDIA's TensorRT support matrix lists FP8 and FP4 as not available on compute capability 8.7, which is Jetson Orin. Use INT8 or a 4-bit weight format such as GGUF Q4 on Orin. Jetson Thor supports FP8 and FP4, but even Whisper large-v3 fits a Thor module at FP16, so those formats are not needed for memory there.

Bottom line

Whisper is a small model by LLM standards, and every size loads on an 8 GB Jetson at Q4. What changes with size is how much memory is left. Small and medium leave room for a camera pipeline or a chat model on an 8 GB module. large-v3 leaves about 1.5 GB at Q4 and needs a 16 GB module to run at FP16. Speed is published only for the smallest models, so measure large-v3 on your own module before relying on it for live transcription.

Size Whisper on your own module, with the memory the rest of your device needs.

Open Model Memory Fit