LLaVA 1.5 7B on Jetson Thor T5000: FITS

Memory breakdown

Weights source

• second-state (community GGUF) · hf:second-state/Llava-v1.5-7B-GGUF · verified 2026-09-07 · source · class A

  llava-v1.5-7b-Q4_K_M.gguf 4.081 GB (published file size)

Headroom

Largest context that fits at 1 sequence: 4096 tokens. Max concurrent sequences at 4096 tokens: 40. Sequence ladder: 28 comfortable (≤60%) · 40 likely viable (≤85%) · fails at 49.

Other quantisations on Jetson Thor T5000

LLaVA 1.5 7B on other modules

Constraints

Sources

• NVIDIA · nvidia.com:jetson-modules-spec-table · verified 2026-09-07 · source · class A

  Jetson Thor T5000: 128 GB

• Hugging Face / lmsys · hf:llava-hf/llava-1.5-7b-hf · config.json · verified 2026-09-07 · source · class A

  layers 32, kv_heads 32, head_dim 128, max context 4096

• second-state (community GGUF) · hf:second-state/Llava-v1.5-7B-GGUF · verified 2026-09-07 · source · class A

  llava-v1.5-7b-Q4_K_M.gguf 4.081 GB (published file size)

• second-state (community GGUF) · hf:second-state/Llava-v1.5-7B-GGUF · verified 2026-09-07 · source · class A

  GGUF mmproj f16 (vision) 0.624 GB

• NVIDIA · nvidia-blog:mastering-llm-techniques-inference-optimization · Key-value caching · verified 2026-09-07 · source · class A

  KV cache size per token = 2 × (num_layers) × (num_heads × dim_head) × precision_in_bytes

• NVIDIA · nvidia-docs:cuda-for-tegra-appnote · Memory Management · verified 2026-09-07 · source · class A

  Jetson iGPU and CPU share the same DRAM: OS, runtime and model all draw on one pool.

Assumptions

• Weights: llava-v1.5-7b-Q4_K_M.gguf: 4.081 GB published file size (class A).
• KV cache: 2 × 32 layers × 32 KV heads × 128 head dim × 4672 tokens × 2 bytes × 1 sequence = 2336 MB (class D formula).
• 1 image × 576 tokens added to the context ((336 / 14)² = 576 patches, no pooling (vision_config image_size 336, patch_size 14); class D).
• Vision tower: GGUF mmproj f16 (vision) 0.624 GB published file size (class A).
• Runtime overhead: llama.cpp / Ollama: 400 MB base + 6% of weights (CUDA context + compute buffer for the prompt batch (n_batch 512)) (class E).
• OS 900 MB (headless) + 10% reserve from the shared memory budget (class E).

Method and limitations

total = weights (published artefact size, else parameters × bytes per parameter) + KV cache (2 × layers × KV heads × head dim × context × bytes × sequences) + vision tower (VLM) + runtime overhead + OS headroom + reserve, on the memory budget shared with Camera Stream Capacity. Verdict: FITS < 85% of module memory ≤ TIGHT < 100% ≤ DOES_NOT_FIT. Vision pipelines take their per-part figures from the memory estimator and are rebuilt on the same budget.

• Weights are class A when a published quantised artefact matches; otherwise parameters × bytes-per-parameter is class D for block quants (Q4, INT8, NVFP4) and class A for fixed-width types (FP16, FP8).
• KV cache, runtime overhead, OS headroom and reserve are class D/E formulas — engineering models, not vendor measurements. Validate on device with sudo tegrastats; llama.cpp prints its own KV and compute-buffer sizes at load.
• Full method, evidence classes and the confidence rule: Model Memory Fit methodology.

Links

• Live engine (this exact workload): https://edgeaistack.ai/engines/model-memory-fit/?mode=vlm&hw=jetson_thor_t5000&model=llava-1.5-7b-hf&quant=q4&ctx=4096&seq=1&rt=llama_cpp&os=headless&mv=1.0&dv=2026-09-07
• Methodology: /methodology/model-memory-fit/
• Dataset: /datasets/model-memory.json
• Hugging Face repository: llava-hf/llava-1.5-7b-hf