Home / Engines / Model Memory Fit

Will this model fit in memory?

Weights, KV cache, runtime overhead and OS headroom evaluated against a Jetson module's memory pool for an LLM, VLM, ASR model or vision pipeline. Weights come from a published GGUF/safetensors file size when one exists, otherwise from the model's own parameter count; the KV cache comes from its own layer, head and head-dimension architecture — nothing is guessed. The result says FITS, TIGHT or DOES NOT FIT, the largest context and concurrency that still fit, and how the next quantisation down compares.

01 · Define workload

Mode & Hardware
Mode
LLM
VLM
ASR
Vision pipeline
Platform
Loading hardware catalog…
Model
Model
Select a platform first
Quantisation
Context tokens
Sequences
KV cache precision
FP16
INT8
Runtime
System
OS
Headless
Desktop
Advanced
Reserve %
Co-located workloads (MB)
Select a platform and a model to continue

03 · How this works

Weights are a published number, not a guess

When the registry has a published quantised file (a GGUF from bartowski/unsloth, or the vendor's own safetensors index) the exact artefact size is used — evidence class A. Otherwise weights are parameters × bytes-per-parameter for that quantisation: fixed widths (FP16, FP8) are class A by definition; block quants (INT8/Q8_0, Q4_K_M, NVFP4) use the median measured-artefact-to-parameter ratio across the registry, class D.

KV cache is the model's own architecture

KV cache = 2 × layers × KV heads × head dimension × context tokens × bytes-per-value × concurrent sequences, per NVIDIA's published inference-optimization formula. Layers, KV heads and head dimension are read from the model's own config.json — not estimated from parameter count — so a Grouped-Query Attention model with few KV heads gets a correspondingly small cache. A Custom model asks for the same four numbers directly.

Runtime overhead, OS headroom, reserve

Runtime overhead is an engineering heuristic per runtime — a fixed base plus a percentage of weights for the CUDA/TVM/PyTorch context and compute buffers — evidence class E, stated on every result. OS headroom and reserve use the same shared memory budget as Camera Stream Capacity and the Memory Estimator, so all three engines agree on what a module "actually" has available. Vision pipeline mode reuses the Memory Estimator's per-part figures (weights, activations, framework, workspace, decode buffers, tracker) and rebuilds them on this same budget.

Full formulas, data tables and what invalidates a result: methodology. Static tables: model memory fit by module and quantisation. Raw dataset: datasets/model-memory.json.

04 · FAQ

Where do the model sizes and architecture numbers come from?
From the publisher's own config.json, safetensors index, or GGUF file listing on Hugging Face — layer count, KV heads, head dimension, max context and published quantised file sizes are all class-A registry data with a source line and URL. When a config.json has no explicit head_dim, it is derived as hidden size ÷ heads and marked as derived.
How is the KV cache size calculated?
2 × layers × KV heads × head dimension × context tokens × bytes-per-value × concurrent sequences, per NVIDIA's own inference-optimization formula. It scales linearly with context length and with how many sequences run at once, which is why long contexts and high concurrency are usually what push a model from FITS to TIGHT.
What do FITS, TIGHT and DOES NOT FIT mean?
FITS is under 85% of the module's memory, TIGHT is 85–100% (it will likely run but leaves no margin and risks swapping), and DOES NOT FIT is at or over 100% of capacity.
Can I check a model that isn't in the registry?
Yes — choose Custom and enter the parameter count, layer count, KV heads and head dimension from the model's own config.json. Weights are then estimated from parameters × bytes-per-parameter (class D) instead of a measured file size.