TinkerLab
ARTIFACT // RUN-LLAMA-CPP-V1-FP8-MOE.SH (SH)

llama.cpp v1.0.0 Hardened Server (OCP FP8 KV Cache, MoE Host Cache & LFM3) Launcher (run-llama-cpp-v1-fp8-moe.sh)

Launches llama-server (v1.0.0+ SemVer) with OCP FP8 (E4M3) KV cache, FlashAttention, configurable MoE CPU host-memory caching, and SSRF/DNS-rebinding-hardened OpenAI-compatible endpoints.

[KEY VERIFIED TAKEAWAY / TL;DR]This reproducible configuration artifact "llama.cpp v1.0.0 Hardened Server (OCP FP8 KV Cache, MoE Host Cache & LFM3) Launcher" (run-llama-cpp-v1-fp8-moe.sh, SH) is paired with Tinker AI Lab's hardware-verified guide "llama-cpp-v1-0-fp8-kv-cache-moe-host-cache-and-sglang-consortium-lfm3-kimi-k31". It exposes 5 customizable browser variables (GGUF_MODEL_PATH=models/LiquidAI_LFM3-1.7B-Q4_K_M.gguf, KV_CACHE_TYPE=fp8_e4m3, CTX_SIZE=32768, N_CPU_MOE=0, SERVER_PORT=8080) with loopback-first security defaults.
01 // HOMELABLocal AI / HomeLab· Verified: 2026-11-16
Browse All 60 Artifacts →
run-llama-cpp-v1-fp8-moe.sh

llama.cpp v1.0.0 Hardened Server (OCP FP8 KV Cache, MoE Host Cache & LFM3) Launcher

Launches llama-server (v1.0.0+ SemVer) with OCP FP8 (E4M3) KV cache, FlashAttention, configurable MoE CPU host-memory caching, and SSRF/DNS-rebinding-hardened OpenAI-compatible endpoints.

View Paired Guide →
Customizable Variables (Saved in Browser)
UTF-8 Encoded · Live Client-Side Compilation
#!/usr/bin/env bash
# Tinker AI — llama.cpp v1.0.0 OCP FP8 KV Cache & MoE Host-Memory Cache Launcher (Guide #51)
set -euo pipefail

MODEL="models/LiquidAI_LFM3-1.7B-Q4_K_M.gguf"
KV_TYPE="fp8_e4m3"
CTX="32768"
CPU_MOE="0"
PORT="8080"

echo "=== Verifying llama.cpp v1.0.0+ binary & starting hardened llama-server ==="
llama-server --version || true

EXTRA_FLAGS=()
if [ "$CPU_MOE" -gt 0 ] 2>/dev/null; then
  EXTRA_FLAGS+=(--n-cpu-moe "$CPU_MOE")
fi

# Note: llama.cpp v1.0.0 enables SSRF/DNS-rebinding hardening & auto-cancels MCP calls on disconnect
exec llama-server \
  -m "$MODEL" \
  -c "$CTX" \
  -fa on \
  --cache-type-k "$KV_TYPE" \
  --cache-type-v "$KV_TYPE" \
  -ngl 99 \
  "${EXTRA_FLAGS[@]}" \
  --host 127.0.0.1 \
  --port "$PORT"
PARAMETER SPECIFICATION // 参数化变量规范说明表

Customizable Parameter Reference for run-llama-cpp-v1-fp8-moe.sh

This SH artifact exposes 5 template variables compiled client-side in your browser.

Template TokenParameter DescriptionVerified DefaultExample Format
{{GGUF_MODEL_PATH}}GGUF Model Path (e.g. LFM3-1.7B or Qwen3.5-35B-A3B)models/LiquidAI_LFM3-1.7B-Q4_K_M.ggufmodels/LiquidAI_LFM3-1.7B-Q4_K_M.gguf
{{KV_CACHE_TYPE}}KV Cache Precision (f16 | q8_0 | fp8_e4m3)fp8_e4m3fp8_e4m3
{{CTX_SIZE}}Context Window (-c)3276832768
{{N_CPU_MOE}}MoE Layers Kept in Host RAM Cache (0 for dense)00
{{SERVER_PORT}}HTTP Server Port80808080
PAIRED HARDWARE MANIFEST // 配套实测环境与边界记录

Verified Environment for "llama.cpp v1.0.0 (OCP FP8 KV Cache & MoE Host-Memory Caching) vs SGLang Consortium: Running Liquid AI LFM3 (256K Ctx) & Kimi-K3.1-Pro Across CUDA, Vulkan, SYCL & Hexagon NPUs"

Tested Hardware & OS

Workstation / Desktop: NVIDIA RTX 5090 (32GB Blackwell), Apple Mac mini M5 Pro (64GB Unified), Intel Arc Pro B60 (24GB SYCL), Snapdragon X2 Elite (OpenCL Adreno / Hexagon NPU) | Cluster Reference: 16×141GB+ H200/B200 or NVLink Fusion GPU+NPU racks for Kimi-K3.1-Pro

Pinned Stack & Model Versions
llama.cpp v1.0.0 & ggml v1.0.0 (SemVer C-API, 35+ legacy llama.h functions pruned, RPC protocol v9)SGLang Consortium stack (Nov 12, 2026) & NVIDIA/Rebellions NVLink Fusion disaggregated inference referenceModels: LiquidAI/LFM3-1.7B (256K ctx, ~1.05 GB Q4), LiquidAI/LFM3-VL-2B (128K ctx), LiquidAI/LFM3-Omni-2B (2.2B), moonshotai/Kimi-K3.1-Pro (2.8T total / 104B active, MXFP4)
Offline Boundary (100% Air-Gapped Local Inference & Hardened llama-server)

Once GGUF/safetensors weights are cached locally, llama-server v1.0.0 runs completely offline with built-in SSRF, CORS, and DNS-rebinding protections.

Known Operational Limits & Failure Modes
  • RPC Protocol v9 Breaking Upgrade: llama.cpp v1.0.0 removed legacy DIAG_MASK_INF / DIAG_MASK_ZERO ops and bumped the ggml-rpc protocol to v9. Every distributed worker node must be upgraded to v1.0.0+ simultaneously—mixing v1.0.0 clients with pre-1.0 rpc-server nodes fails immediately.
  • 35+ Pruned C-API Symbols in llama.h: Downstream C/C++ bindings (custom Rust/Python/Go wrappers) calling deprecated functions like llama_set_warmup or legacy defragmentation-threshold setters will fail to compile against libllama v1.0.0 until updated to the clean 1.x header.
  • Kimi-K3.1-Pro Is Strictly Multi-Node Cluster Territory (~1.56 TB MXFP4): Even with native MXFP4 weights and MXFP8 activations, Kimi-K3.1-Pro (2.8T total, 104B active across 93 layers) spans 96 safetensors shards (~1.56 TB)—requiring 16× 141GB+ HBM GPUs or high-RAM multi-node NVLink/RDMA clusters rather than a single workstation.
DEPLOYMENT CHECKLIST // 配置落地核验步骤

3-Step Verification Checklist After Deploying run-llama-cpp-v1-fp8-moe.sh

  1. Verified llama-server --version is v1.0.0+ and upgraded all distributed rpc-server nodes together to RPC protocol v9.
  2. Enabled -fa on --cache-type-k fp8_e4m3 --cache-type-v fp8_e4m3 (or q8_0) for 32K+ context workloads.
  3. Bound llama-server to 127.0.0.1 behind a reverse proxy (never exposing raw --host 0.0.0.0 without auth) to complement v1.0.0 SSRF/DNS-rebinding protections.