SC26 Disaggregated On-Premise Inference & PCIe Gen5 KV Offload: MiTAC G4520G6 + 8× Skymizer HTX301-4S (2TB Decode-First LPU for Kimi K3.1 2.8T) vs TRUSTA T7P5 AI Scaler 3-Tier vLLM
Deep dive into two SC26 (November 2026) on-premise architectures breaking the HBM cost wall for multi-trillion-parameter models and long-context agents: MiTAC G4520G6 air-cooled 4U decode servers packing 8× Skymizer HTX301-4S cards (2TB commodity DRAM @ ~155W/card, LISA v3) paired with GPU prefill nodes to run 2.8T Kimi K3.1 Pro MXFP4 (~1.56TB), versus ADATA TRUSTA AI Scaler Toolkit + T7P5 PCIe Gen5 NVMe (13.5 GB/s, <7W, FDP) delivering reboot-persistent 3-tier GPU -> DRAM -> SSD vLLM KV cache offloading.
MiTAC G4520G6 4U (Xeon 6 + 8× Skymizer HTX301-4S · 2TB DRAM) + MI350X Prefill Node vs 96GB GPU + 256GB DRAM + TRUSTA T7P5 PCIe Gen5 NVMe (U.2/E3.S/E1.S · 13.5 GB/s)
Both disaggregated Prefill/Decode LAN transfers (RoCEv2/RDMA) and 3-tier TRUSTA PCIe Gen5 NVMe KV cache persistence operate purely inside your local rack or workstation with zero cloud API calls.
- Skymizer HTX301-4S is a "decode-first" LPU using commodity DRAM instead of HBM—running heavy 64K+ prompt prefill directly on HTX301 cards yields poor Time-To-First-Token (TTFT); it must be paired with a compute-dense GPU prefill node (e.g., MI350X / H200 / RTX PRO 6000) via high-speed RDMA KV transfer.
- Each HTX301-4S PCIe card exposes 4 independent HTX301 PCIe devices (32 logical endpoints across an 8-card MiTAC G4520G6 chassis), requiring sufficient host PCIe bifurcation/switch topology and LISA v3 tensor-parallel sharding.
- TRUSTA AI Scaler / LMCache 3-tier NVMe KV offloading speeds up repeated long-context prefixes and multi-turn agent cache hits across reboots, but it does NOT substitute for GPU VRAM needed to hold active model weights—and consumer QLC SSDs will quickly throttle or wear out under continuous KV write amplification without enterprise 3-DWPD / FDP drives.
1. Why SC26 Split Inference in Half: MiTAC G4520G6 + 8× Skymizer HTX301-4S (2TB) for 2.8T Kimi K3.1 Pro
~1.56 TB in native MXFP4) traditionally required two liquid-cooled 8x H200/B200 HBM racks ($500K+). At SC26 (November 2026), MiTAC Computing and Skymizer demonstrated a far cheaper two-server disaggregated on-premise architecture:
- Compute-Bound Prefill Server (MiTAC G8825Z5): Houses AMD EPYC CPUs + AMD Instinct MI350X/MI355X (or high-FLOPS GPUs) exclusively to digest long input prompts and generate initial KV caches at maximum matrix-FLOPS.
- Memory-Bound Decode Server (MiTAC G4520G6 4U Air-Cooled): Pairs Intel Xeon 6 host CPUs with 8× Skymizer HTX301-4S PCIe cards. Each ~155W card carries 4× HyperThought HTX301 decode-first LPU chips (LISA v3 instruction set) surrounded by 256 GB of commodity high-capacity DRAM—delivering 2 TB of card-attached decode memory in a standard air-cooled 4U chassis. Once the KV cache streams over RDMA from the prefill server, the 8x HTX301-4S pool holds all 1.56 TB of Kimi K3.1 Pro weights plus ~350+ GB of active KV cache locally without a single gigabyte of scarce HBM.2. TRUSTA AI Scaler Toolkit + T7P5 PCIe Gen5 NVMe (13.5 GB/s, <7W, FDP): 3-Tier Persistent vLLM KV Cache
GPU VRAM -> Host DRAM -> PCIe Gen5 NVMe): Active decode stays in GPU VRAM; warm sessions spill to host DRAM; cold multi-turn agent threads and 128K RAG documents persist onto TRUSTA T7P5 / T7P51 ECO PCIe Gen5 x4 SSDs (13,500 MB/s seq read, 10,300 MB/s seq write, 3.2M random read IOPS, <7W active controller power, ~3 DWPD with NVMe FDP — Flexible Data Placement to slash write amplification).
- Survives Server Reboots & AMX Prefill Boost: Because KV blocks persist on NVMe, restarting vLLM or resuming an agent session the next morning reloads cached KV tensors at 13.5 GB/s instead of burning tens of seconds recomputing 100K-token prompts. Pairing the toolkit’s Strata engine with Intel AMX adds up to +25% host prefill acceleration ([Spec-Est.] marks spec-derived projections below):| Architecture & Hardware | Total Usable Memory & Power | Best-Fit Workload & Bottleneck | Key Deployment Constraint |
|---|---|---|---|
| MiTAC G4520G6 + 8× Skymizer HTX301-4S (4U Decode) | 2 TB card DRAM (~1.92 TB usable) · ~155W/card (Air-cooled) | Hosts 2.8T Kimi K3.1 Pro MXFP4 (1.56 TB) locally [Spec-Est.] | Must pair with GPU prefill server (RDMA KV link) + LISA v3 runtime |
Single Skymizer HTX301-4S (256GB PCIe LPU) | 256 GB DRAM (~240 GB usable) · ~155W PCIe | Qwen3-235B-A22B / Nemotron 3.5 Super 120B decode [Spec-Est.] | Exposes 4 PCIe devices per card; avoid heavy prefill on LPU alone |
| 96GB GPU + 256GB DRAM + TRUSTA T7P5 Gen5 SSD (3-Tier vLLM) | 96GB VRAM + 256GB DRAM + TB-scale 13.5 GB/s SSD (<7W/drive) | Multi-user 64K–128K RAG/Agents; reboot-persistent KV (5–10x TTFT hit [Spec-Est.]) | Weights still require GPU VRAM; use 3-DWPD / FDP enterprise NVMe |
| Traditional 16× H200 / B200 HBM Cluster (2 Nodes) | 2.25–2.88 TB HBM3e · 15–24 kW (Liquid/High-CFM Rack) | Fastest combined prefill + decode throughput (no disagg overhead) | Extreme CapEx (5–10x hardware cost) & facility power/cooling limits |
3. Reproducible Commands: Deploy 3-Tier vLLM NVMe KV Offload & Audit Disaggregated Prefill/Decode Nodes
vLLM v0.33.0+ with LMCacheConnectorV1 (mirroring the TRUSTA AI Scaler 3-tier hierarchy) to offload cold KV blocks onto a dedicated PCIe Gen5 NVMe mount, or split NixlConnector into kv_producer (GPU prefill) and kv_consumer (decode node) across your local RDMA network.# 1) Verify PCIe Gen5 x4 link speed (32GT/s) & Intel AMX BF16 support
sudo lspci -vv | grep -E 'Non-Volatile|LnkSta:' | head -n 12
grep -m1 -o 'amx_bf16' /proc/cpuinfo || true
# 2) Launch vLLM with 3-tier persistent KV offload (GPU -> 192GB DRAM -> TRUSTA T7P5 Gen5 NVMe)
export LMCACHE_USE_EXPERIMENTAL=True
export LMCACHE_LOCAL_CPU=True LMCACHE_MAX_LOCAL_CPU_SIZE=192
export LMCACHE_LOCAL_DISK="file:///mnt/trusta-t7p5-kv/" LMCACHE_MAX_LOCAL_DISK_SIZE=1024
vllm serve Qwen/Qwen3-235B-A22B-FP8 --max-model-len 131072 --enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' --port 8100
# 3) On a MiTAC G4520G6 decode node: confirm all 32x HTX301 PCIe endpoints (8 cards x 4 chips)
lspci | grep -ciE 'Skymizer|HTX301'Frequently Asked Questions & Edge Cases
Q: Can TRUSTA AI Scaler / PCIe Gen5 SSD KV offloading let a 24GB GPU run a 120B model that does not fit in VRAM?
No—keep weight residency and KV cache tiering strictly separate. PCIe Gen5 SSDs (13.5 GB/s) are great for saving and reloading multi-turn / multi-agent KV caches so you never recompute 64K–128K prompts when switching users or restarting the server. Active model weights are read on every single token, so streaming 70 GB of weights from a 13.5 GB/s SSD would crawl at <0.2 tok/s. Weights must still sit in GPU VRAM, unified memory, or high-channel DDR5/MRDIMM.
Q: Why does Skymizer HTX301-4S use 4 chips per card with commodity DRAM instead of one giant HBM die?
Token generation (decode) for trillion-parameter MoEs like Kimi K3.1 Pro (1.56 TB MXFP4) is starved for memory capacity per dollar and per watt, not peak FP16 matrix FLOPS. Splitting a card into 4x independent HTX301 decode engines surrounded by dense commodity DRAM reaches 256 GB per ~155W PCIe card (2 TB in one air-cooled 4U chassis) without CoWoS/HBM packaging bottlenecks—leaving the expensive high-FLOPS GPU strictly for the short, compute-intensive prefill burst.
SC26 Disaggregated Inference (Skymizer HTX301-4S 2TB) & TRUSTA PCIe Gen5 3-Tier vLLM KV Offload Launcher
Audits PCIe Gen5 NVMe bandwidth / FDP namespaces for TRUSTA AI Scaler 3-tier vLLM KV offloading (GPU -> DRAM -> NVMe) with persistent cross-reboot context reload, or orchestrates disaggregated Prefill (MI350X/GPU) + Decode (MiTAC G4520G6 + 8x Skymizer HTX301-4S 2TB) endpoints.
#!/usr/bin/env bash
# Tinker AI — SC26 Disaggregated Prefill/Decode (Skymizer HTX301-4S 2TB) & TRUSTA PCIe Gen5 3-Tier vLLM KV Offload (Guide #59)
set -euo pipefail
MODE="trusta_tiered_kv_vllm"
MODEL="Qwen/Qwen3-235B-A22B-FP8"
KV_DIR="/mnt/trusta-t7p5-kv"
DRAM_GB="192"
PORT="8100"
echo "=== [SC26 Storage & Accelerator Audit] Checking PCIe Gen5 NVMe Links & CPU AMX ==="
lspci -vv 2>/dev/null | grep -E 'Non-Volatile|LnkSta:' | head -n 16 || true
grep -m1 -o 'amx_bf16' /proc/cpuinfo && echo "[OK] Intel AMX BF16 detected (accelerates host prefill up to ~25%)" || echo "[INFO] Non-AMX CPU detected"
if [ "$MODE" = "trusta_tiered_kv_vllm" ]; then
echo "=== [TRUSTA AI Scaler / 3-Tier vLLM KV Pool] GPU VRAM -> ${DRAM_GB}GB DRAM -> $KV_DIR (PCIe Gen5 NVMe) ==="
mkdir -p "$KV_DIR"
df -h "$KV_DIR"
export LMCACHE_USE_EXPERIMENTAL=True
export LMCACHE_LOCAL_CPU=True
export LMCACHE_MAX_LOCAL_CPU_SIZE="$DRAM_GB"
export LMCACHE_LOCAL_DISK="file://$KV_DIR/"
export LMCACHE_MAX_LOCAL_DISK_SIZE=1024
exec vllm serve "$MODEL" \
--max-model-len 131072 \
--enable-prefix-caching \
--gpu-memory-utilization 0.90 \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' \
--host 0.0.0.0 --port "$PORT"
elif [ "$MODE" = "disagg_prefill_node" ]; then
echo "=== [Disaggregated Prefill Node] Serving Compute-Bound Prefill Producer on Port $PORT ==="
exec vllm serve "$MODEL" \
--max-model-len 131072 \
--gpu-memory-utilization 0.92 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' \
--host 0.0.0.0 --port "$PORT"
elif [ "$MODE" = "disagg_decode_htx301" ]; then
echo "=== [MiTAC G4520G6 + 8x Skymizer HTX301-4S] Auditing 32x PCIe Decode Endpoints (4x HTX301 per Card) ==="
HTX_COUNT="$(lspci | grep -ciE 'Skymizer|HTX301' || true)"
echo "Detected HTX301 PCIe endpoints: $HTX_COUNT (expected: 4 per HTX301-4S card, 32 for 8-card 2TB decode node)"
echo "[INFO] Launch LISA v3 decode consumer runtime pointing at prefill producer port $PORT for model $MODEL"
echo "Example: lisa-serve --role kv_consumer --model '$MODEL' --kv-port '$PORT' --mem-budget-gb 1920"
else
echo "Unknown RUN_MODE=$MODE (expected: trusta_tiered_kv_vllm | disagg_prefill_node | disagg_decode_htx301)" >&2
exit 1
fi