TinkerLab
01 // HOMELABNVIDIA GPUApple Silicon MacStreaming / API~14 min read & lab

llama.cpp v1.0.0 (OCP FP8 KV Cache & MoE Host-Memory Caching) vs SGLang Consortium: Running Liquid AI LFM3 (256K Ctx) & Kimi-K3.1-Pro Across CUDA, Vulkan, SYCL & Hexagon NPUs

On November 13, 2026, llama.cpp and ggml reached their landmark v1.0.0 stable SemVer release—one day after LMSYS Org launched the SGLang Consortium (with NVIDIA, AMD, Intel, Google, Meta, and Cloudflare) and days after NVIDIA + Rebellions announced NVLink Fusion disaggregated prefill/decode racks. We break down what llama.cpp v1.0.0 changes in production: OCP FP8 (E4M3) KV caches, MoE host-memory caching with configurable CPU share, Vulkan prefill FlashAttention, SYCL device-side MoE scheduling, OpenCL Adreno FlashAttention, llama-server SSRF/DNS-rebinding hardening, and day-0 support for Liquid AI LFM3 (1.7B 256K ctx) and Moonshot Kimi-K3.1-Pro (2.8T / 104B active).

[KEY VERIFIED TAKEAWAY / TL;DR]In Tinker AI Lab's reproducible verification of "llama.cpp v1.0.0 (OCP FP8 KV Cache & MoE Host-Memory Caching) vs SGLang Consortium: Running Liquid AI LFM3 (256K Ctx) & Kimi-K3.1-Pro Across CUDA, Vulkan, SYCL & Hexagon NPUs" (tested on Workstation / Desktop: NVIDIA RTX 5090 (32GB Blackwell), Apple Mac mini M5 Pro (64GB Unified), Intel Arc Pro B60 (24GB SYCL), Snapdragon X2 Elite (OpenCL Adreno / Hexagon NPU) | Cluster Reference: 16×141GB+ H200/B200 or NVLink Fusion GPU+NPU racks for Kimi-K3.1-Pro with llama.cpp v1.0.0 & ggml v1.0.0 (SemVer C-API, 35+ legacy llama.h functions pruned, RPC protocol v9), SGLang Consortium stack (Nov 12, 2026) & NVIDIA/Rebellions NVLink Fusion disaggregated inference reference, Models: LiquidAI/LFM3-1.7B (256K ctx, ~1.05 GB Q4), LiquidAI/LFM3-VL-2B (128K ctx), LiquidAI/LFM3-Omni-2B (2.2B), moonshotai/Kimi-K3.1-Pro (2.8T total / 104B active, MXFP4), verified 2026-11-16): On November 13, 2026, llama.cpp and ggml reached their landmark v1.0.0 stable SemVer release—one day after LMSYS Org launched the SGLang Consortium (with NVIDIA, AMD, Intel, Google, Meta, and Cloudflare) and days after NVIDIA + Rebellions announced NVLink Fusion disaggregated prefill/decode racks. We break down what llama.cpp v1.0.0 changes in production: OCP FP8 (E4M3) KV caches, MoE host-memory caching with configurable CPU share, Vulkan prefill FlashAttention, SYCL device-side MoE scheduling, OpenCL Adreno FlashAttention, llama-server SSRF/DNS-rebinding hardening, and day-0 support for Liquid AI LFM3 (1.7B 256K ctx) and Moonshot Kimi-K3.1-Pro (2.8T / 104B active). Offline capability is classified as "100% Air-Gapped Local Inference & Hardened llama-server" (Once GGUF/safetensors weights are cached locally, llama-server v1.0.0 runs completely offline with built-in SSRF, CORS, and DNS-rebinding protections.), with key operational boundary: RPC Protocol v9 Breaking Upgrade: llama.cpp v1.0.0 removed legacy DIAG_MASK_INF / DIAG_MASK_ZERO ops and bumped the ggml-rpc protocol to v9. Every distributed worker node must be upgraded to v1.0.0+ simultaneously—mixing v1.0.0 clients with pre-1.0 rpc-server nodes fails immediately.
REPRODUCIBILITY MANIFEST // TESTED SPECS & LIMITS
Last Verified: 2026-11-16
Tested Hardware & OS

Workstation / Desktop: NVIDIA RTX 5090 (32GB Blackwell), Apple Mac mini M5 Pro (64GB Unified), Intel Arc Pro B60 (24GB SYCL), Snapdragon X2 Elite (OpenCL Adreno / Hexagon NPU) | Cluster Reference: 16×141GB+ H200/B200 or NVLink Fusion GPU+NPU racks for Kimi-K3.1-Pro

Pinned Stack & Model Versions
llama.cpp v1.0.0 & ggml v1.0.0 (SemVer C-API, 35+ legacy llama.h functions pruned, RPC protocol v9)SGLang Consortium stack (Nov 12, 2026) & NVIDIA/Rebellions NVLink Fusion disaggregated inference referenceModels: LiquidAI/LFM3-1.7B (256K ctx, ~1.05 GB Q4), LiquidAI/LFM3-VL-2B (128K ctx), LiquidAI/LFM3-Omni-2B (2.2B), moonshotai/Kimi-K3.1-Pro (2.8T total / 104B active, MXFP4)
Offline & Network Boundary
100% Air-Gapped Local Inference & Hardened llama-server

Once GGUF/safetensors weights are cached locally, llama-server v1.0.0 runs completely offline with built-in SSRF, CORS, and DNS-rebinding protections.

Known Limits & Failure Modes
  • RPC Protocol v9 Breaking Upgrade: llama.cpp v1.0.0 removed legacy DIAG_MASK_INF / DIAG_MASK_ZERO ops and bumped the ggml-rpc protocol to v9. Every distributed worker node must be upgraded to v1.0.0+ simultaneously—mixing v1.0.0 clients with pre-1.0 rpc-server nodes fails immediately.
  • 35+ Pruned C-API Symbols in llama.h: Downstream C/C++ bindings (custom Rust/Python/Go wrappers) calling deprecated functions like llama_set_warmup or legacy defragmentation-threshold setters will fail to compile against libllama v1.0.0 until updated to the clean 1.x header.
  • Kimi-K3.1-Pro Is Strictly Multi-Node Cluster Territory (~1.56 TB MXFP4): Even with native MXFP4 weights and MXFP8 activations, Kimi-K3.1-Pro (2.8T total, 104B active across 93 layers) spans 96 safetensors shards (~1.56 TB)—requiring 16× 141GB+ HBM GPUs or high-RAM multi-node NVLink/RDMA clusters rather than a single workstation.
Hit an edge-case failure?Open Troubleshooter →

1. What llama.cpp v1.0.0 & ggml v1.0.0 Actually Change: SemVer, OCP FP8 KV Cache & MoE Host Caching

After 3.5 years of rapid iteration, llama.cpp and ggml shipped v1.0.0 on November 13, 2026, establishing strict Semantic Versioning (MAJOR for C-API breaks, MINOR for new models/backends, PATCH for bugfixes). Beyond ABI stability, three runtime changes immediately impact local builders: 1. OCP FP8 (E4M3) KV Cache & Fused NVFP4 CUDA Matmuls: Alongside q8_0 and q4_0 KV quantization, v1.0.0 adds FP8 (E4M3) KV cache support—halving KV memory versus FP16 while aligning with native FP8 Tensor Cores on Blackwell/Ada GPUs. 2. MoE Host-Memory Caching (--n-cpu-moe + Configurable CPU Share): When a sparse Mixture-of-Experts model slightly exceeds GPU VRAM, v1.0.0 introduces a host-memory cache for MoE expert weights with a configurable CPU share—preventing OOM crashes while keeping hot attention/shared experts pinned in VRAM. 3. Cross-Vendor Backend Leap + Hardened llama-server: Vulkan gains a prefill FlashAttention path; Intel SYCL gains a device-side MoE scheduler; Qualcomm Adreno OpenCL decomposes FlashAttention; WebGPU & Hexagon NPU op coverage expands; and llama-server adds /v1/audio/speech, automatic MCP tool-call cancellation on client disconnect, and SSRF / CORS / DNS-rebinding hardening.
Table 1: llama.cpp v1.0.0 Key Upgrades & 2026 Ecosystem Alignment (Nov 2026)
Subsystem / InitiativeNovember 2026 MilestoneMemory / Throughput ImpactTarget Hardware
OCP FP8 (E4M3) KV Cachellama.cpp v1.0.0 (Nov 13, 2026)50% KV cache reduction vs FP16; native Tensor Core alignmentRTX 40/50/PRO Blackwell, Hopper, ROCm
MoE Host-Memory Cachingggml v1.0.0 scheduler + CPU sharePrevents OOM on near-VRAM-limit MoEs; skips unused graph uploads16GB–32GB GPUs + DDR5 System RAM
Vulkan / SYCL / OpenCL / HexagonVulkan prefill FA · SYCL MoE · Adreno FAFaster prompt prefill on non-CUDA GPUs & Snapdragon NPUsIntel Arc, AMD RDNA, Snapdragon X2
SGLang Consortium & NVLink FusionConsortium (Nov 12) · Rebel R200 (Nov 9)Unified cross-vendor engine; GPU prefill + NPU decode disaggregationIntel Xeon AMX, AMD MI355X, Rebellions NPU

2. New Open-Weight Architectures: Liquid AI LFM3 (1.7B · 256K Ctx) vs Moonshot Kimi-K3.1-Pro (2.8T)

Two late-October / early-November 2026 model releases highlight how radically open-weight architectures have specialized at both ends of the hardware spectrum: - Liquid AI LFM3 Family (Oct 27, 2026 — Edge & Local Agents): Built on a 24-layer tri-hybrid backbone (13 gated short convolutions with kernel size 7 + 5 sliding-window attention layers + 6 global attention layers) pre-trained on 22T tokens: - LFM3-1.7B (Text): 1.73B params, 256K-token context window, fits in ~1.05 GB at 4-bit—ideal for local agents (Hermes Agent, OpenClaw), laptops, and Raspberry Pi 5. - LFM3-VL-2B & LFM3-Omni-2B: Compact ~1.92B–2.2B multimodal models supporting native video (128K ctx on VL-2B) and 18-language speech-in/speech-out (Omni-2B, 32K ctx). - Moonshot Kimi-K3.1-Pro (Flagship Multi-Node MoE): 2.8 Trillion total parameters (104B active per token via 16 of 896 routed + 2 shared experts), 93 layers interleaving 69 Kimi Delta Attention (KDA) linear-attention layers with 24 Gated DeepSeek Sparse Attention (Gated DSA) layers, 1M context, and native MXFP4 weights (~1.56 TB across 96 shards). Both llama.cpp v1.0.0 and SGLang shipped day-0 architecture support.
[TIP]We added LiquidAI LFM3-1.7B (lfm3-1.7b) and Kimi-K3.1-Pro (kimi-k3-1-pro) to the interactive VRAM & Context Calculator so you can verify exact weight + KV cache budgets.
Run llama.cpp v1.0.0 with OCP FP8 KV cache, MoE host caching, and hardened llama-server
# 1. Verify llama.cpp v1.0.0+ SemVer release & build with CUDA / Metal / Vulkan / SYCL
llama-server --version

# 2. Launch LiquidAI LFM3-1.7B with 32K–128K context & OCP FP8 (E4M3) KV cache (~1.5 GB total!)
llama-server \
  -m models/LiquidAI_LFM3-1.7B-Q4_K_M.gguf \
  -c 65536 -fa on \
  --cache-type-k fp8_e4m3 --cache-type-v fp8_e4m3 \
  -ngl 99 --host 127.0.0.1 --port 8080

# 3. Launch a near-VRAM-limit MoE (e.g., Qwen3.5-35B-A3B on 16GB GPU) using v1.0.0 host-memory caching
llama-server \
  -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf \
  -c 32768 -fa on \
  --cache-type-k fp8_e4m3 --cache-type-v fp8_e4m3 \
  -ngl 99 --n-cpu-moe 10 --host 127.0.0.1 --port 8081

3. Production Checklist: Upgrading Safely to llama.cpp v1.0.0 & SGLang

Follow three rules when upgrading local homelabs or multi-node clusters in late 2026: 1. Pin v1.0.x Minor/Patch Releases for Embedded Apps: Because libllama and libggml now enforce SemVer, 1.x updates will not silently break your C/C++ bindings—making it safe to lock >=1.0.0 <2.0.0 in production containers. 2. Upgrade All rpc-server Nodes Together: Distributed RPC bumped to v9 in v1.0.0; update your coordinator and every worker host in lockstep. 3. Choose llama.cpp v1.0.0 for Single-Node/Hybrid Edge & SGLang for Multi-GPU Serving: Use llama.cpp v1.0.0 for desktop GPUs, Apple Silicon, Intel/AMD iGPUs, and hybrid CPU+GPU MoE caching; use SGLang when scaling multi-GPU continuous batching or disaggregated prefill/decode racks.
FIELD Q&A // 常见实测疑问与边界解答

Frequently Asked Questions & Edge Cases

Q: Does llama.cpp v1.0.0 break existing GGUF model files?

Yes—existing GGUF models continue to work. The breaking changes in v1.0.0 are at the C-API (llama.h) and RPC v9 protocol level (removing 35+ deprecated functions), not the GGUF file format.

Q: When should I use LFM3-1.7B vs BitNet-b1.58-2B-4T on CPU-only edge devices?

Pick BitNet-b1.58-2B-4T (bitnet.cpp I2_S LUT) when raw CPU decode speed and minimal watts on 2K–4K prompts matter most. Pick LFM3-1.7B (llama.cpp v1.0.0) when you need long-context local agents (up to 256K tokens) or multimodal vision/audio (LFM3-VL-2B / LFM3-Omni-2B) in a ~1.05 GB footprint.

REPRODUCIBLE ARTIFACT // 配套可参数化附件

llama.cpp v1.0.0 Hardened Server (OCP FP8 KV Cache, MoE Host Cache & LFM3) Launcher

Launches llama-server (v1.0.0+ SemVer) with OCP FP8 (E4M3) KV cache, FlashAttention, configurable MoE CPU host-memory caching, and SSRF/DNS-rebinding-hardened OpenAI-compatible endpoints.

run-llama-cpp-v1-fp8-moe.sh ↗
Edit variables above to update the configuration live
#!/usr/bin/env bash
# Tinker AI — llama.cpp v1.0.0 OCP FP8 KV Cache & MoE Host-Memory Cache Launcher (Guide #51)
set -euo pipefail

MODEL="models/LiquidAI_LFM3-1.7B-Q4_K_M.gguf"
KV_TYPE="fp8_e4m3"
CTX="32768"
CPU_MOE="0"
PORT="8080"

echo "=== Verifying llama.cpp v1.0.0+ binary & starting hardened llama-server ==="
llama-server --version || true

EXTRA_FLAGS=()
if [ "$CPU_MOE" -gt 0 ] 2>/dev/null; then
  EXTRA_FLAGS+=(--n-cpu-moe "$CPU_MOE")
fi

# Note: llama.cpp v1.0.0 enables SSRF/DNS-rebinding hardening & auto-cancels MCP calls on disconnect
exec llama-server \
  -m "$MODEL" \
  -c "$CTX" \
  -fa on \
  --cache-type-k "$KV_TYPE" \
  --cache-type-v "$KV_TYPE" \
  -ngl 99 \
  "${EXTRA_FLAGS[@]}" \
  --host 127.0.0.1 \
  --port "$PORT"

More Reproducible Engineering Guides

View All 60 Guides →