Local LLM VRAM & Context Budget Calculator
Model weight file size does not equal runtime memory budget. Calculate the exact sum of Quantized Weights + Dynamic KV Cache + Compute Buffers against real hardware limits, and export a ready-to-run Modelfile.
1. Model, Quantization & Hardware
Comfortable sweet spot for 14B Q4_K_M/Q5_K_M models with 16K–32K context.
High-accuracy structured output, multi-doc RAG, and complex instruction following
Best balance of speed, memory, and perplexity; standard default in Ollama.
Fits Comfortably: Full GPU/Metal Acceleration
- Weight file is 9.18 GB, but runtime adds 1.61 GB KV Cache (14% of total) + 0.86 GB compute buffer = 11.65 GB total.
- Fits comfortably inside Apple Silicon Mac (24GB Unified) (65% of 18 GB usable budget). Full GPU/Metal acceleration active.
# Generated by Tinker AI VRAM & Context Budget Calculator FROM qwen2.5:14b-q4_k_m PARAMETER num_ctx 8192 PARAMETER temperature 0.3
# Ollama runtime environment variables OLLAMA_NUM_PARALLEL=1 OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=f16
Verified Hardware Usable Limits & Recommended Model Configurations
All configurations below stay 100% inside GPU VRAM / Metal unified memory without triggering SSD swap or PCIe shared-memory fallback.
| Hardware Profile | Usable Ceiling | Sweet-Spot Model & Quant | Recommended Context & KV | Measured Speed |
|---|---|---|---|---|
| Raspberry Pi 5 (8GB) | 6.2 GB | Qwen2.5-3B (Q4_K_M) | 2K–4K (f16) | 8.4–9.6 tok/s |
| Pi 5 AI HAT+ 2 (Hailo-10H) / Jetson Super (8GB) | 6.4–6.8 GB | Qwen2.5-1.5B / Llama-3.2-3B / Qwen3-4B | 4K–8K (q8_0 KV) | 19–34 tok/s |
| RTX 4060 / 3070 (8GB VRAM) | 7.1 GB | Qwen3-8B / Llama-3.1-8B (Q4_K_M) | 8K–16K (q8_0 KV) | 55–68 tok/s |
| Intel Arc B580 / RTX 4070 (12GB) | 10.6–10.9 GB | Qwen3-8B (Q8_0) / Qwen3-14B (Q4_K_M) | 8K–16K (q8_0 KV) | 38–55 tok/s |
| Mac M2/M3/M4 (16GB–18GB) | 11.5–13.2 GB | Qwen2.5-14B (Q4_K_M) | 16K–32K (q8_0 KV) | 22–31 tok/s |
| RTX 5080 / 5070 Ti / 5060 Ti / RX 9070 XT (16GB) | 14.6–14.8 GB | Qwen3-14B / Qwen2.5-Coder-14B (Q4_K_M / Q5_K_M) | 16K–32K (q8_0 KV) | 39–79 tok/s |
| Mac mini M4 Pro (24GB · 273 GB/s) | 18.2 GB | Qwen2.5-14B / R1-14B (Q4_K_M / Q5_K_M) | 16K–32K (q8_0 KV) | 31–38 tok/s |
| Intel Arc Pro B60 / RTX 3090 / 4090 (24GB) | 22.0–22.6 GB | Qwen3-30B-A3B MoE / Qwen2.5-32B (Q4_K_M) | 16K (q8_0 KV) | 21–86 tok/s |
| RTX 5090 / PRO 4500 / R9700 (32GB) | 30.0–30.2 GB | Qwen3-32B / Qwen2.5-VL-32B (Q5_K_M / Q6_K) | 32K–64K (q8_0 KV) | 26–78 tok/s |
| Mac Max (36GB–48GB Unified) | 27.5–37.0 GB | Qwen2.5-Coder-32B / R1-32B (Q5_K_M) | 32K (q8_0 KV) | 18–26 tok/s |
| RTX PRO 5000 (72GB) / PRO 6000 (96GB ECC) | 68.5–91.5 GB | Llama-3.3-70B (Q6_K/FP8) / Qwen3-Next-80B-A3B | 64K–128K (q8_0 KV) | 20–148 tok/s |
| M5 Max / DGX Spark / Thor / Strix Halo (128GB) | 96.0–110 GB | Qwen3-30B-A3B MoE (Q8_0) / Llama-3.3-70B | 32K–64K (q8_0 KV) | 61–112 tok/s (MoE) |
| Mac Studio (64GB–128GB) / Dual 3090 / Dual B60 | 44.5–104 GB | Llama-3.3-70B (Q4_K_M / Q8_0) | 32K–64K (q8_0 KV) | 12–24 tok/s |
| M3 Ultra (256G/512G) / Dual Spark / EPYC Hybrid | 220–480 GB | Qwen3-235B-A22B / DeepSeek-R1-671B (4-bit) | 32K–64K (q8_0 KV) | 14–38 tok/s (MoE) |
Why GGUF File Size ≠ Actual Runtime VRAM
Total runtime memory is the sum of three distinct allocations. Ignoring KV cache and compute buffers is the #1 cause of sudden OOM spillover:
1. Weights: Params_B × (Bits_Per_Weight / 8) × 1.03
2. KV_Cache: (num_ctx / 1024) × KV_Rate_Per_1K × KV_Quant × NUM_PARALLEL
3. Buffer: 0.45 GB + (0.022 × Params_B) + (num_ctx / 16384) × 0.18 GB scratchpad
Common VRAM Sizing & KV Cache Questions
Does q8_0 KV cache hurt output quality?
Across 8K–32K benchmarks, q8_0 (8-bit) KV cache shows <0.2% perplexity delta vs f16 while cutting context memory in half.
Why does a 16GB Mac only have ~11.5GB usable?
macOS caps default Metal wired memory at ~70–75% of unified RAM to protect WindowServer and kernel buffers from SSD swap thrashing.