The Edge LLM Runtime Stack 2026: llama.cpp, Ollama, TensorRT Edge-LLM, ExecuTorch, vLLM, MLX, LiteRT-LM

Why Runtime Choice Actually Matters

The hardware conversation around edge AI gets most of the attention — Jetson Thor, Hailo-8, Coral, Apple Neural Accelerators. The runtime conversation gets almost none of it. That's a mistake. The runtime is the layer that determines whether your 70B model actually hits production latency, whether it survives a JetPack upgrade, and whether you can port to a second platform without rewriting your serving code.

Three reasons it matters more than people assume:

1. Throughput swings of 3× are normal. The same Llama 3.3 70B model on the same Jetson Thor moves dramatically between llama.cpp (GGUF Q4_K_M), TensorRT-LLM with NVFP4, and TensorRT-LLM with EAGLE-3 speculative decoding. A 2.5× uplift from speculative decoding alone is documented on NVIDIA's blog. None of that is silicon; it's all runtime.
2. Quantization formats don't port cleanly. A GGUF Q4_K_M file is portable across CPUs, Jetsons, and Apple Silicon — but you'll leave Blackwell-specific NVFP4 throughput on the table. A model converted to NVFP4 won't run on Orin. ExecuTorch .pte files won't run in llama.cpp. Plan for this at design time, not deployment time.
3. KV cache management is where production deployments live or die. For long-context workloads, the KV cache often exceeds model weight memory. Runtimes differ wildly in how they handle it — PagedAttention (vLLM), KV cache quantization (TensorRT-LLM, llama.cpp), and attention-sink eviction (StreamingLLM, DuoAttention) are all on the table.

The Seven Runtime Profiles

llama.cpp — CPU king, max portability

The de facto standard for CPU-based LLM inference, evolved from a proof-of-concept into a core production tool. C++ with extensive SIMD optimization. Runs on x86, ARM, Apple Silicon, even Raspberry Pi. GGUF is now the lingua franca for quantized model distribution on Hugging Face — hundreds of community model ports use it.

Ollama — developer experience and prototyping

A lightweight Go wrapper around llama.cpp with HTTP API, model management, and a one-line install. Ollama crossed 165k GitHub stars in 2026 because ollama run deepseek-r1 just works. Most developer prototypes and self-hosted AI stacks (Open WebUI, Dify, n8n) sit on top of Ollama rather than llama.cpp directly.

NVIDIA TensorRT Edge-LLM SDK — Jetson production runtime

New in JetPack 7.1 (Jan 2026). An open-source C++ runtime built specifically for Jetson-class devices that operate under tight memory budgets, hard latency constraints, and shared GPU/CPU pressure from perception and control workloads. Typical flow: export PyTorch model → ONNX → TensorRT optimization → deploy engine. Supports NVFP4, FP8, W4A16, and EAGLE-3 speculative decoding on Thor.

Meta ExecuTorch — PyTorch-native for mobile and embedded

Meta's production runtime for on-device PyTorch, hit v1.0 in late 2025. 50KB base footprint — runs on microcontrollers through flagship smartphones. Twelve hardware backends. Now powers AI features in Instagram, WhatsApp, Messenger, and Facebook at billions-of-users scale. Around 80% of the most popular edge LLMs on Hugging Face have working ExecuTorch exports.

vLLM — edge server, multi-tenant

Server-class LLM serving with PagedAttention — the algorithm that 2–3×'d throughput by treating KV cache like virtual memory pages. Originally a datacenter project, but increasingly used on edge servers and even on Jetson Thor for multi-user scenarios. NVIDIA's Cat AI Assistant demo at CES 2026 used vLLM serving Qwen3 4B on Jetson Thor.

Apple MLX — Apple Silicon only

Apple's array framework optimized for M-series unified memory architecture. NumPy-like API in Python, with a Swift binding for native iOS/macOS apps. MLX-VLM extends it to vision-language models. On an M3 Max with 64GB unified memory, a 4-bit quantized Llama 3.1 70B runs at ~8 tokens/sec — practical for local development.

Google LiteRT-LM — Android / iOS mobile

Google's successor to TFLite, tuned for running Gemma-class small models on phones. The Google AI Edge Gallery reference app on Play Store and App Store is built on LiteRT-LM — open-source codebase that shows the full model-management, inference, and agentic-tool-call pipeline for production mobile AI. Real-world performance: a small Gemma variant generates draft text on an iPhone in airplane mode in a few seconds.

Hardware × Runtime Pairing Matrix

The right runtime is almost always determined by the hardware you've already chosen. Use this as a sanity check:

Three Mistakes We See Most Often

1. Picking a runtime before picking the model

Teams pick "we'll use llama.cpp" before knowing what model they'll deploy, then end up needing a multimodal model that's only well-supported in ExecuTorch, or a 70B that needs NVFP4 via TensorRT. The runtime should come after model selection. Pick the model your task needs, then pick the runtime that runs it best on your hardware.

2. Ignoring the KV cache for long-context workloads

The KV cache grows linearly with sequence length and can exceed model weight memory for long-context inference. For RAG, agent workflows, or document summarization at the edge, KV cache management is often more impactful than weight quantization. Use a runtime that supports KV cache quantization (TensorRT-LLM does; llama.cpp partially does) or chunked attention (StreamingLLM-style approaches, now in vLLM).

3. Treating "it runs in Ollama on my laptop" as a deployment plan

Ollama is excellent for prototyping. It's not a production runtime for embedded devices. The latency overhead, the always-on HTTP server, the model-management daemon — none of it survives a real industrial deployment. Prototype in Ollama, then port to llama.cpp directly or to a platform-native runtime (TensorRT Edge-LLM on Jetson, ExecuTorch on mobile, MLX on Apple) before you ship.

What We Expect to Change in 2026

• llama.cpp Vulkan backend matures — making it the default cross-vendor GPU path and reducing CUDA's lock-in for community models.
• ExecuTorch picks up share on Jetson — as the PyTorch export workflow tightens, expect ExecuTorch to compete with TensorRT Edge-LLM for non-NVFP4 workloads.
• vLLM lands on smaller hardware — work on a "vLLM Edge" variant for single-device, low-concurrency edge servers is already in flight in the community.
• 1-bit and BitNet runtimes go from research to early production — particularly for ultra-low-power deployments where weight memory dominates power draw.

Frequently Asked Questions