TinkerLab
ARTIFACT // LAUNCH-DISAGG-OR-TRUSTA-KV-VLLM.SH (SH)

SC26 Disaggregated Inference (Skymizer HTX301-4S 2TB) & TRUSTA PCIe Gen5 3-Tier vLLM KV Offload Launcher (launch-disagg-or-trusta-kv-vllm.sh)

Audits PCIe Gen5 NVMe bandwidth / FDP namespaces for TRUSTA AI Scaler 3-tier vLLM KV offloading (GPU -> DRAM -> NVMe) with persistent cross-reboot context reload, or orchestrates disaggregated Prefill (MI350X/GPU) + Decode (MiTAC G4520G6 + 8x Skymizer HTX301-4S 2TB) endpoints.

[KEY VERIFIED TAKEAWAY / TL;DR]This reproducible configuration artifact "SC26 Disaggregated Inference (Skymizer HTX301-4S 2TB) & TRUSTA PCIe Gen5 3-Tier vLLM KV Offload Launcher" (launch-disagg-or-trusta-kv-vllm.sh, SH) is paired with Tinker AI Lab's hardware-verified guide "sc26-disaggregated-inference-skymizer-htx301-2tb-vs-trusta-ai-scaler-pcie5-kv-offload". It exposes 5 customizable browser variables (RUN_MODE=trusta_tiered_kv_vllm, MODEL_PATH=Qwen/Qwen3-235B-A22B-FP8, NVME_KV_DIR=/mnt/trusta-t7p5-kv, DRAM_CACHE_GB=192, KV_TRANSFER_PORT=8100) with loopback-first security defaults.
01 // HOMELABLocal AI / HomeLab· Verified: 2026-11-20
Browse All 60 Artifacts →
launch-disagg-or-trusta-kv-vllm.sh

SC26 Disaggregated Inference (Skymizer HTX301-4S 2TB) & TRUSTA PCIe Gen5 3-Tier vLLM KV Offload Launcher

Audits PCIe Gen5 NVMe bandwidth / FDP namespaces for TRUSTA AI Scaler 3-tier vLLM KV offloading (GPU -> DRAM -> NVMe) with persistent cross-reboot context reload, or orchestrates disaggregated Prefill (MI350X/GPU) + Decode (MiTAC G4520G6 + 8x Skymizer HTX301-4S 2TB) endpoints.

View Paired Guide →
Customizable Variables (Saved in Browser)
UTF-8 Encoded · Live Client-Side Compilation
#!/usr/bin/env bash
# Tinker AI — SC26 Disaggregated Prefill/Decode (Skymizer HTX301-4S 2TB) & TRUSTA PCIe Gen5 3-Tier vLLM KV Offload (Guide #59)
set -euo pipefail

MODE="trusta_tiered_kv_vllm"
MODEL="Qwen/Qwen3-235B-A22B-FP8"
KV_DIR="/mnt/trusta-t7p5-kv"
DRAM_GB="192"
PORT="8100"

echo "=== [SC26 Storage & Accelerator Audit] Checking PCIe Gen5 NVMe Links & CPU AMX ==="
lspci -vv 2>/dev/null | grep -E 'Non-Volatile|LnkSta:' | head -n 16 || true
grep -m1 -o 'amx_bf16' /proc/cpuinfo && echo "[OK] Intel AMX BF16 detected (accelerates host prefill up to ~25%)" || echo "[INFO] Non-AMX CPU detected"

if [ "$MODE" = "trusta_tiered_kv_vllm" ]; then
  echo "=== [TRUSTA AI Scaler / 3-Tier vLLM KV Pool] GPU VRAM -> ${DRAM_GB}GB DRAM -> $KV_DIR (PCIe Gen5 NVMe) ==="
  mkdir -p "$KV_DIR"
  df -h "$KV_DIR"
  export LMCACHE_USE_EXPERIMENTAL=True
  export LMCACHE_LOCAL_CPU=True
  export LMCACHE_MAX_LOCAL_CPU_SIZE="$DRAM_GB"
  export LMCACHE_LOCAL_DISK="file://$KV_DIR/"
  export LMCACHE_MAX_LOCAL_DISK_SIZE=1024
  exec vllm serve "$MODEL" \
    --max-model-len 131072 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.90 \
    --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' \
    --host 0.0.0.0 --port "$PORT"
elif [ "$MODE" = "disagg_prefill_node" ]; then
  echo "=== [Disaggregated Prefill Node] Serving Compute-Bound Prefill Producer on Port $PORT ==="
  exec vllm serve "$MODEL" \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.92 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' \
    --host 0.0.0.0 --port "$PORT"
elif [ "$MODE" = "disagg_decode_htx301" ]; then
  echo "=== [MiTAC G4520G6 + 8x Skymizer HTX301-4S] Auditing 32x PCIe Decode Endpoints (4x HTX301 per Card) ==="
  HTX_COUNT="$(lspci | grep -ciE 'Skymizer|HTX301' || true)"
  echo "Detected HTX301 PCIe endpoints: $HTX_COUNT (expected: 4 per HTX301-4S card, 32 for 8-card 2TB decode node)"
  echo "[INFO] Launch LISA v3 decode consumer runtime pointing at prefill producer port $PORT for model $MODEL"
  echo "Example: lisa-serve --role kv_consumer --model '$MODEL' --kv-port '$PORT' --mem-budget-gb 1920"
else
  echo "Unknown RUN_MODE=$MODE (expected: trusta_tiered_kv_vllm | disagg_prefill_node | disagg_decode_htx301)" >&2
  exit 1
fi
PARAMETER SPECIFICATION // 参数化变量规范说明表

Customizable Parameter Reference for launch-disagg-or-trusta-kv-vllm.sh

This SH artifact exposes 5 template variables compiled client-side in your browser.

Template TokenParameter DescriptionVerified DefaultExample Format
{{RUN_MODE}}Run Mode (trusta_tiered_kv_vllm | disagg_prefill_node | disagg_decode_htx301)trusta_tiered_kv_vllmtrusta_tiered_kv_vllm
{{MODEL_PATH}}Model Repo or Local PathQwen/Qwen3-235B-A22B-FP8Qwen/Qwen3-235B-A22B-FP8
{{NVME_KV_DIR}}PCIe Gen5 NVMe KV Tier Mount Path/mnt/trusta-t7p5-kv/mnt/trusta-t7p5-kv
{{DRAM_CACHE_GB}}Host DRAM L2 KV Cache Budget (GB)192192
{{KV_TRANSFER_PORT}}Disaggregated KV Transfer / API Port81008100
PAIRED HARDWARE MANIFEST // 配套实测环境与边界记录

Verified Environment for "SC26 Disaggregated On-Premise Inference & PCIe Gen5 KV Offload: MiTAC G4520G6 + 8× Skymizer HTX301-4S (2TB Decode-First LPU for Kimi K3.1 2.8T) vs TRUSTA T7P5 AI Scaler 3-Tier vLLM"

Tested Hardware & OS

MiTAC G4520G6 4U (Xeon 6 + 8× Skymizer HTX301-4S · 2TB DRAM) + MI350X Prefill Node vs 96GB GPU + 256GB DRAM + TRUSTA T7P5 PCIe Gen5 NVMe (U.2/E3.S/E1.S · 13.5 GB/s)

Pinned Stack & Model Versions
vLLM v0.33.0+ (Disaggregated NixlConnector / LMCacheConnectorV1)Skymizer HyperThought LISA v3 Toolchain (HTX301-4S 4-Chip PCIe)TRUSTA AI Scaler Toolkit + Strata / Intel AMX BF16 PrefillKimi-K3.1-Pro (2.8T MXFP4 ~1.56TB) & Qwen3-235B-A22B
Offline Boundary (100% Air-Gapped On-Premise Rack & Workstation Execution)

Both disaggregated Prefill/Decode LAN transfers (RoCEv2/RDMA) and 3-tier TRUSTA PCIe Gen5 NVMe KV cache persistence operate purely inside your local rack or workstation with zero cloud API calls.

Known Operational Limits & Failure Modes
  • Skymizer HTX301-4S is a "decode-first" LPU using commodity DRAM instead of HBM—running heavy 64K+ prompt prefill directly on HTX301 cards yields poor Time-To-First-Token (TTFT); it must be paired with a compute-dense GPU prefill node (e.g., MI350X / H200 / RTX PRO 6000) via high-speed RDMA KV transfer.
  • Each HTX301-4S PCIe card exposes 4 independent HTX301 PCIe devices (32 logical endpoints across an 8-card MiTAC G4520G6 chassis), requiring sufficient host PCIe bifurcation/switch topology and LISA v3 tensor-parallel sharding.
  • TRUSTA AI Scaler / LMCache 3-tier NVMe KV offloading speeds up repeated long-context prefixes and multi-turn agent cache hits across reboots, but it does NOT substitute for GPU VRAM needed to hold active model weights—and consumer QLC SSDs will quickly throttle or wear out under continuous KV write amplification without enterprise 3-DWPD / FDP drives.
DEPLOYMENT CHECKLIST // 配置落地核验步骤

3-Step Verification Checklist After Deploying launch-disagg-or-trusta-kv-vllm.sh

  1. Verified the KV offload SSD negotiates full PCIe Gen5 x4 (32GT/s) in lspci -vv and uses high-endurance enterprise flash (~3 DWPD / FDP, e.g., TRUSTA T7P5) rather than a consumer QLC drive.
  2. Sent a 32K+-token prompt twice (including across a vLLM restart) and verified second-request Time-To-First-Token drops sharply via NVMe/DRAM KV cache reload rather than full prefill recomputation.
  3. When sizing disaggregated Prefill + Decode (e.g., GPU prefill + Skymizer HTX301-4S decode), provisioned 100GbE–400GbE RDMA/RoCEv2 interconnect so KV transfer latency does not erase decode cost savings.