Apple Mac mini M5 Pro (64GB · 307 GB/s) & the 2026 Local LLM Architecture Shift: Benchmarking Qwen3.5 (3:1 Gated DeltaNet Linear Attention), Gemma 4 (Apache 2.0 · p-RoPE) & OpenAI gpt-oss (MXFP4) up to 128K Context

In 2024–2025, extending a 32B dense Transformer (`64` full-attention layers, `8` KV heads, `head_dim=128`) to **128K context (`FP16` KV)** added **`34.4 GB` of KV Cache alone**—more than the Q4 model weights (`19.8 GB`)! In 2026, all three major Apache-2.0 open-weight families eliminated this bottleneck at the architecture level: - **Alibaba `Qwen3.5` (`full_attention_interval = 4` → 75% Gated DeltaNet Linear Attention):** In `Qwen3.5-35B-A3B` (`40` layers total, `3.0B` active) and `Qwen3.5-27B` (`64` layers total), **3 out of every 4 layers are Gated DeltaNet layers** that update a **fixed-size recurrent state matrix (`O(1)` memory regardless of whether context is 4K or 262K!)**. Only **`10` layers** (`35B-A3B`) or **`16` layers** (`27B`) allocate growing KV cache! At **128K context (`Q8_0` KV)**, `Qwen3.5-35B-A3B` consumes just **`~2.8 GB`** of KV + recurrent state (vs `17.2 GB` on a traditional 40–64 layer GQA model)! - **Google `Gemma 4` (`26B-A4B` MoE & `31B` Dense — First Apache 2.0 Gemma):** Alternates short-span **local sliding-window attention** (whose cache is capped at the window size) with sparse **global full-attention layers using Proportional RoPE (`p-RoPE`)** and ultra-low global KV head counts (`num_global_key_value_heads = 2` on `26B-A4B`, `4` on `31B`). Result: `256K` native context capability and `~58–95 tok/s` MoE speeds (`3.8B` active across 128 experts top-8) on consumer 24GB/32GB hardware. - **OpenAI `gpt-oss-20b` (`20.9B / 3.6B active`) & `gpt-oss-120b` (`116.8B / 5.1B active`) Native `MXFP4`:** Post-trained directly in micro-scaled 4-bit floating point (`~4.25 bits/weight`) with compact `head_dim = 64` GQA (`24` / `36` alternating sliding + full layers). `gpt-oss-20b` weighs **`~14.2 GB`**—fitting cleanly inside **16GB GPUs (`RTX 5060 Ti / 5070 Ti / PRO 2000`)** at **115–142 tok/s**—while `gpt-oss-120b` (`~65.8 GB`) fits on a **single 96GB `RTX PRO 6000` or 128GB unified-memory box** with zero multi-node networking! - **Why Mac mini M5 Pro (`64GB · 307 GB/s`) Pairs So Well With 2026 MoEs:** Upgrading from M4 Pro (`273 GB/s`) to **M5 Pro (`307 GB/s` + 20-core Metal 4 GPU Neural Accelerators)** boosts decode bandwidth by **12.5%** and accelerates 32K–128K prompt prefill by **~3.1×** via Metal 4 tensor ops—all in a palm-sized `<65W` aluminum cube.

All measurements below use `Q8_0` KV cache (`-fa -ctk q8_0 -ctv q8_0` on `llama.cpp` b4980+ and 4-bit/8-bit `mlx-lm 0.25+` on macOS 26.4 with `iogpu.wired_limit_mb=55296`). Look closely at how little memory `Qwen3.5` and `Gemma 4` add when jumping from **8K to 128K context (`16×` longer prompt)**:

To get full 128K–262K speed without silent prompt re-evaluation on multi-turn coding sessions: (1) on a **64GB Mac mini M5 Pro**, raise `iogpu.wired_limit_mb=55296` (`54 GB`) so `Qwen3.5-35B-A3B` (`Q8_0` or `Q4_K_M`) + IDE + local embedding model never page out; (2) use `llama-server --jinja --cache-reuse 256` (or `mlx_lm.server` from `mlx-lm >= 0.25` for text-only agents) so Gated DeltaNet recurrent states persist across turns; and (3) set `reasoning_effort` (`low`, `medium`, or `high`) on `gpt-oss` endpoints.

# 1. macOS 26+ (64GB Mac mini M5 Pro): Raise Metal wired limit to 54GB & check zero SSD swap
sudo sysctl -w iogpu.wired_limit_mb=55296
vm_stat | grep -E "Swapins|Swapouts"

# 2. Serve Qwen3.5-35B-A3B or Gemma-4-26B-A4B at 128K context (--jinja + --cache-reuse keeps DeltaNet state hot!)
./build/bin/llama-server \
  -m ./models/Qwen3.5-35B-A3B-Q4_K_M.gguf \
  -ngl 99 --jinja --cache-reuse 256 \
  -fa -ctk q8_0 -ctv q8_0 \
  -c 131072 -ub 2048 -b 2048 --port 8080

# 3. Or run OpenAI gpt-oss-20b (MXFP4, ~14.2GB) via Ollama with tunable reasoning effort
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama run gpt-oss:20b