How the Model Memory Fit engine decides what fits.

Contents

1. What it computes, and what it does not
2. Inputs and defaults
3. Weights
4. KV cache
5. VLM image tokens and vision tower
6. Runtime overhead
7. OS headroom and reserve
8. Vision-pipeline mode
9. Verdict thresholds and constraint statuses
10. Evidence classes and the confidence rule
11. Model registry
12. Module memory
13. Documents
14. What invalidates a result
15. Limitations and gaps
16. Method changelog

1. What it computes, and what it does not

The engine builds a memory budget for one model on one Jetson module at one quantisation, context length, concurrency and runtime, and compares the total against the module's installed memory. Four modes share the same budget: llm (weights + KV cache + runtime overhead + OS + reserve), vlm (llm plus a vision tower and image tokens added to the context), asr (Whisper-style encoder–decoder, a small decoder KV cache), and vision (detector pipelines, which delegate their per-part figures to the Memory Estimator and rebuild them on this same budget so the two engines agree on OS headroom and reserve).

It does not measure your actual runtime's memory use, predict tokens/s, model KV-cache eviction or paging strategies, or account for a container runtime's own overhead beyond the stated runtime constant. Where the registry has no data for a field the engine falls back to a stated formula (never a silent guess) and labels the result's evidence class accordingly.

2. Inputs and defaults

Every input has a stated default so a minimal request (platform + model) still returns a result.

3. Weights

The engine prefers a measured artefact: if the registry has a published quantised file for this model and quant (a GGUF from bartowski, unsloth or ggml-org, or a safetensors shard set), its published size is the weight figure, class A. Otherwise weights are parameters × bytes-per-parameter.

bytes_per_param is a fixed storage width for FP16/BF16 (2 bytes) and FP8 (1 byte) — class A, true by definition. For block quants (INT8/GGUF Q8_0, Q4_K_M, NVFP4) the nominal bit width understates the real size because each block also stores a scale, so the engine uses the median measured artefact-size ÷ parameters ratio across the registry's own rows for that quant — class D, a EdgeAIStack-derived figure, not a vendor one.

Source for the Q8_0 block layout: ggml-org (llama.cpp) · github:ggml-org/llama.cpp/ggml-common.h §block_q8_0 · verified 2026-09-07 — “QK8_0 32 … block_q8_0 { ggml_half d; int8_t qs[QK8_0]; }”.

4. KV cache

The KV cache formula is the standard transformer key-value cache size, applied per sequence and summed across concurrent sequences. ASR mode caps the token count at the decoder's text limit (448 tokens for Whisper), since the encoder side has no growing cache.

Source: NVIDIA · nvidia-blog:mastering-llm-techniques-inference-optimization §Key-value caching · verified 2026-09-07 — “KV cache size per token = 2 × (num_layers) × (num_heads × dim_head) × precision_in_bytes”. Class A: the formula is the architecture-derived cache size; class D overall because layers/kv_heads/head_dim for a given model may themselves be class-A config.json fields or, when config.json omits head_dim, derived as hidden_size ÷ num_attention_heads (flagged per-model as head_dim_derived).

5. VLM image tokens and vision tower

A VLM adds two things beyond the text backbone: image tokens appended to the context (so the KV cache grows), and the vision tower's own weights.

image_tokens_per_image is model-specific and stated on each model's page: for example Qwen2.5-VL uses (H/28)×(W/28) tokens after a 2×2 spatial merge (1280 assumed for a ~1 MP image); Gemma 3 pools its SigLIP patches to a fixed 256 tokens per image. Vision-tower parameters, when not directly stated, are estimated with the standard ViT parameter count 12 × layers × hidden², kept in FP16 regardless of the text quantisation (vision towers are rarely quantised below FP16 in practice).

6. Runtime overhead

A fixed base plus a percentage of weights, per runtime — engineering heuristics (class E), not vendor-published figures. These cover the CUDA context, compute buffers and (for llama.cpp/MLC) the paged KV allocator's bookkeeping beyond the KV bytes already counted in §4.

runtime_mb = base_mb + workspace_pct × (weights_mb + vision_mb) / 100.

7. OS headroom and reserve

Shared with Camera Stream Capacity so the two engines never disagree about what the OS costs.

Jetson modules share one memory pool between CPU and GPU (class A: NVIDIA · nvidia-docs:cuda-for-tegra-appnote §Memory Management · verified 2026-09-07 — “the iGPU and the CPU share the same SoC DRAM”), so the OS, the runtime and the model all draw on the same total; there is no separate VRAM.

8. Vision-pipeline mode

For detector pipelines (YOLO, ResNet and similar), the engine does not duplicate the weight/activation/workspace model — it calls the Memory Estimator for those parts (weights, activations, framework overhead, decode buffers, tracker memory, secondary models) and rebuilds the total on this engine's shared budget function, so OS headroom and reserve are computed identically to every other mode. This keeps the two engines from silently disagreeing on a workload that could be scored by either.

9. Verdict thresholds and constraint statuses

The memory figure is also reported as a constraint object on the shared V2 framework (status PASS below 80%, NEAR_LIMIT from 80%, FAIL at 100%) — a slightly stricter early-warning line than the FITS/TIGHT/DOES_NOT_FIT verdict, which is tuned for the headline sentence. Two additional constraint types can appear: a software constraint (class A, HIGH confidence) fails when the requested context exceeds the model's max_position_embeddings, and a software constraint reports UNSUPPORTED when runtime is llama_cpp and the registry notes that no GGUF has been published for the model (VILA 1.5 3B and 8B, as of the current dataset).

10. Evidence classes and the confidence rule

Every figure carries one of five evidence classes.

Confidence: a result is MEDIUM when the weight figure came from a published artefact (class A) and module capacity is known; otherwise LOW. This is deliberately conservative — every result mixes at least one class D/E part (KV cache, runtime, OS, reserve), so nothing from this engine is ever labelled HIGH. The software constraint for a context beyond max_position_embeddings is HIGH confidence on its own (it is a direct comparison against a class-A config field), but does not raise the overall result above MEDIUM.

11. Model registry

22 rows: 10 LLM, 9 VLM, 3 ASR. Every field is copied from the publisher's config.json, safetensors index or GGUF repository listing; gated repos (meta-llama, google) are read from a byte-identical community mirror (unsloth, NousResearch) and the mirror is named in the source. Full detail per model, including quantised artefact sizes and Jetson tokens/s measurements: /model-fit/.

12. Module memory

Installed memory per Jetson module, from NVIDIA's technical specifications table (class A). jetson_orin_nano_super is an alias of jetson_orin_nano (same silicon, same memory) and is not listed separately.

13. Documents

14. What invalidates a result

• A new model release or GGUF re-quantisation. Every result URL carries mv (method version) and dv (dataset version); when either differs from the current values the result page says the data has changed since the link was made.
• A different runtime than assumed. The runtime-overhead table is an engineering heuristic (§6); a real llama.cpp build with a larger n_batch, or vLLM's pre-allocated gpu_memory_utilization, can use meaningfully more or less than the stated base + percentage.
• Container and driver overhead. The OS headroom figure (§7) is a bare-JetPack baseline; a Docker/L4T container runtime, X11/Wayland desktop session or other resident services add to it. The coexisting_mb input exists for exactly this.
• Context beyond max_position_embeddings. The engine still computes a KV cache for it and flags a class-A software constraint, but tokens/s and quality beyond the trained context are not modelled at all.
• No GGUF for llama.cpp. VILA 1.5 3B and 8B have no published GGUF as of this dataset; the engine reports UNSUPPORTED for that runtime rather than a number.
• Vision-tower and image-token estimates. Any model whose vision tower has no published mmproj/file size uses the class-D 12×L×H² estimate (§5), which can be off by 20–30% against the model's real parameter count.

15. Limitations and gaps

• No Jetson-measured memory rows yet. The registry's jetson_measurements carry tokens/s (mostly class C, from Jetson AI Lab archives), not measured RAM footprints; every memory total on this site is computed from the formulas above, not read off a running board. Validate with sudo tegrastats.
• Gated repositories. meta-llama and google repos on Hugging Face return 401 for unauthenticated reads; the registry uses byte-identical community mirrors (unsloth, NousResearch) instead and names the mirror in the source.
• VLM parameter counts include the vision tower. A model card's headline size (e.g. “Qwen2.5-VL 3B”) usually names the text backbone only; the registry's params_b is the full safetensors total, so it reads higher than the marketing name. The engine separates text and vision parameters internally (§3, §5) so weight pricing is not double-counted.
• Non-Jetson platforms. The engine falls back to the Memory Estimator's platform table for module capacity (class D) when a platform is not in the Model Memory Fit module registry; that path is LOW confidence.
• No accuracy or quality modelling. A quantisation that fits says nothing about output quality at that quant; that tradeoff is out of scope for this engine.

16. Method changelog

Method changes bump the method version; registry updates (new models, new measurements, corrected sources) bump the dataset version. Both appear in the citation block under every result and in the provenance object of every API response.