SC26 Desktop Cluster & NVMe KV Offload: NVIDIA DGX Spark 4-Node Switchless 200GbE Ring (512GB) + LMCache GPUDirect Storage Running Mistral Large 4 (1.05T) & 700B NVFP4
Architecture, routing, and storage-disaggregation guide for the SC26 NVIDIA DGX Spark HPC Developer Kit: wire 4x GB10 nodes in a switchless dual-port ConnectX-7 200GbE ring (512GB coherent memory, 20TB NVMe), prevent NCCL non-adjacent ring deadlocks, offload multi-turn KV caches via LMCache + cuFile GPUDirect Storage, and size 552B–744B NVFP4 models vs Mistral Large 4 (1.05T, 50B active).
4x NVIDIA DGX Spark HPC Developer Kit (4x GB10 Grace Blackwell · 512GB Coherent LPDDR5X · 20TB NVMe · Dual-Port ConnectX-7 200GbE Switchless Ring)
Mistral Large 4 uses the Mistral Ecosystem Open-Weight License (v26.10), which permits on-premise commercial deployment and fine-tuning while restricting direct hosted MaaS resale without a commercial agreement.
- In a 4-node switchless ring (Node 1 <-> Node 2 <-> Node 3 <-> Node 4 <-> Node 1), diagonally opposite nodes have no direct cable; leaving NCCL on default Tree mode or omitting /30 static routes causes all_reduce hangs or silent fallback to 1GbE/10GbE LAN.
- Mistral Large 4 (~1.05T total, ~50B active + 1.6B vision encoder) exceeds 512GB in 4-bit (~600+ GB with KV), requiring 3.1-bit IQ3_XXS across the 4-node 512GB ring or a 1TB+ AMD EPYC SP7 server for full 4-bit Q4_K_M.
- GPUDirect Storage (cuFile / nvidia_fs) zero-copy NVMe KV offloading requires raw/ext4/XFS mounts without OS encryption layers and IOMMU passthrough compatibility.
1. SC26 Switchless 4-Node 200GbE Ring: 512GB Desktop Cluster Without a $10K Switch
GB10) beyond 2 nodes required an expensive enterprise 200GbE QSFP112 switch. At SC26 (November 2026), NVIDIA introduced the DGX Spark HPC Developer Kit, wiring 4x DGX Spark units directly into a closed loop using both ports of each unit’s integrated dual-port ConnectX-7 (200Gb/s per port) NIC: Node 1 P1 -> Node 2 P0, Node 2 P1 -> Node 3 P0, Node 3 P1 -> Node 4 P0, and Node 4 P1 -> Node 1 P0.
Assigning isolated /30 point-to-point subnets (10.10.12.0/30, 10.10.23.0/30, 10.10.34.0/30, 10.10.41.0/30) with MTU 9000 jumbo frames pools 512GB of coherent LPDDR5X memory (~1.09 TB/s aggregate memory bandwidth across TP=4) and 20TB of NVMe storage right on an office desk under ~1,000W total wall power. Because Node 1 cannot reach Node 3 in a single hop, you must set NCCL_ALGO=Ring and NCCL_CROSS_NIC=1 for GPU collectives while enabling net.ipv4.ip_forward=1 with static diagonal routes so Ray (6379) control-plane heartbeats never spill onto Wi-Fi or 1GbE LAN.2. 512GB Ring Cluster Model Matrix: 700B NVFP4 vs Mistral Large 4 (1.05T)
~472 GB usable cluster memory after OS and Ray/NCCL buffers (~118 GB per node after flushing Linux UMA page cache), hardware-native Blackwell NVFP4 unlocks 550B–744B frontier MoEs, while Mistral Large 4 (~1.05T total, 49–52B active + 1.6B native vision encoder, 1M context) defines the boundary where 3.1-bit quantization or a 1TB EPYC SP7 server becomes necessary ([Spec-Est.] denotes bandwidth- and fabric-derived estimates):| Model (Total / Active) | Quantization & Footprint | 4x DGX Spark Ring (512GB · TP=4) | Multi-Turn 128K TTFT (w/ LMCache GDS) |
|---|---|---|---|
DeepSeek-V4.1-Flash (552B / 8B–16B · MIT) | NVFP4 (~312 GB w/ 64K KV) | Fits cleanly (~42–52 tok/s [Spec-Est.]) | ~1.6s vs ~13.5s cold prefill |
GLM-5.3 (744B / 40B · 1M ctx) | NVFP4 (~432 GB w/ 32K FP8 KV) | Tight fit (~19–24 tok/s [Spec-Est.], flush UMA cache) | ~2.2s vs ~18.8s cold prefill |
Mistral Large 4 (~1.05T / 50B · 1M ctx) | IQ3_XXS (~445 GB) / Q4_K_M (~665 GB) | IQ3_XXS via llama.cpp RPC (~14–18 tok/s); Q4_K_M needs 1TB SP7 | ~2.9s (NVMe KV slab hit) |
3. Hierarchical NVMe KV-Cache Offloading with LMCache + cuFile GPUDirect Storage
700B NVFP4 models on a 512GB ring, memory headroom left for KV cache shrinks to 35–120 GB—meaning concurrent coding agents evict prefix caches constantly.
Integrating LMCache (LMCacheConnectorV1) with NVIDIA GPUDirect Storage (cuFile / nvidia_fs) streams evicted KV blocks directly between GPU/coherent memory and the kit’s 20TB PCIe NVMe pool via zero-copy DMA, bypassing CPU bounce buffers and cutting 128K multi-turn Time-To-First-Token (TTFT) by 5x–8x.# 1) On Node 1 (example): set MTU 9000 on both ConnectX-7 ports & add diagonal route to Node 3
sudo ip link set dev enP7s7f0 mtu 9000 up
sudo ip link set dev enP7s7f1 mtu 9000 up
sudo ip addr add 10.10.12.1/30 dev enP7s7f1 # Link to Node 2 (10.10.12.2)
sudo ip addr add 10.10.41.2/30 dev enP7s7f0 # Link to Node 4 (10.10.41.1)
sudo sysctl -w net.ipv4.ip_forward=1
sudo ip route replace 10.10.23.0/30 via 10.10.12.2 dev enP7s7f1
# 2) Validate switchless 4-node ring collectives (must force NCCL_ALGO=Ring!)
mpirun -np 4 -H 10.10.12.1:1,10.10.12.2:1,10.10.23.2:1,10.10.34.2:1 \
-x NCCL_SOCKET_IFNAME=enP7s7f0,enP7s7f1 -x NCCL_ALGO=Ring -x NCCL_CROSS_NIC=1 \
/opt/nccl-tests/build/all_reduce_perf -b 8M -e 1G -f 2 -g 1
# 3) Flush UMA cache & serve TP=4 NVFP4 with LMCache GPUDirect Storage KV offload
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
LMCACHE_GDS_PATH=/nvme/lmcache_gds LMCACHE_CUFILE_BUFFER_SIZE=4096 \
vllm serve deepseek-ai/DeepSeek-V4.1-Flash-NVFP4 \
--tensor-parallel-size 4 --distributed-executor-backend ray \
--quantization nvfp4 --max-model-len 131072 \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'Frequently Asked Questions & Edge Cases
Q: Why does vLLM TP=4 hang during initialization on a 4-node DGX Spark ring without a switch?
Two culprits cause 95% of hangs: first, NCCL defaults to Tree or direct all-to-all peer assumptions that fail when Node 1 and Node 3 lack a direct cable—fix this with NCCL_ALGO=Ring and NCCL_CROSS_NIC=1. Second, Ray requires full-mesh TCP reachability between all worker IPs; enable net.ipv4.ip_forward=1 and add static /30 routes through adjacent nodes so diagonal TCP packets route cleanly over the 200GbE ring.
Q: Does NVMe KV-cache offloading speed up single-prompt token generation (decode tok/s)?
No—and confusing the two leads to bad hardware builds. LMCache + GDS NVMe offloading dramatically accelerates multi-turn Time-To-First-Token (TTFT) by reloading pre-computed KV tensors at 12–25+ GB/s instead of recomputing attention over 128K tokens. Active per-token decode still runs out of fast GPU/UMA memory bandwidth.
DGX Spark 4-Node Switchless 200GbE Ring (512GB) + LMCache NVMe GDS Launcher
Audits dual-port ConnectX-7 200GbE ring MTU 9000 & IP forwarding, flushes GB10 UMA page cache, locks NCCL_ALGO=Ring to prevent non-adjacent deadlocks, and starts vLLM TP=4 with LMCache GPUDirect Storage (cuFile) NVMe KV offloading.
#!/usr/bin/env bash
# Tinker AI — SC26 DGX Spark 4-Node Switchless 200GbE Ring + LMCache GDS Launcher (Guide #55)
set -euo pipefail
IFACES="enP7s7f0,enP7s7f1"
HEAD_IP="10.10.12.1"
MODEL="deepseek-ai/DeepSeek-V4.1-Flash-NVFP4"
KV_DIR="/nvme/lmcache_gds"
MAX_GB="512"
echo "=== [1/4] Auditing ConnectX-7 200GbE Ring MTU 9000, Diagonal IP Forwarding & GDS ==="
IFS=',' read -ra DEV_ARR <<< "$IFACES"
for dev in "${DEV_ARR[@]}"; do
ip -br link show "$dev" || echo "WARN: Ring interface $dev not found"
done
sudo sysctl -w net.ipv4.ip_forward=1 >/dev/null
lsmod | grep -q nvidia_fs && echo "OK: nvidia_fs (GPUDirect Storage cuFile) kernel module loaded" || echo "INFO: nvidia_fs not loaded; LMCache will fall back to POSIX O_DIRECT"
echo "=== [2/4] Flushing GB10 Coherent UMA Page Cache Before Sharded Weight mmap ==="
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
mkdir -p "$KV_DIR"
echo "=== [3/4] Locking Switchless 4-Node Ring NCCL & LMCache GDS Environment ==="
export NCCL_SOCKET_IFNAME="$IFACES"
export UCX_NET_DEVICES="${DEV_ARR[0]}"
export NCCL_ALGO=Ring
export NCCL_CROSS_NIC=1
export NCCL_BUFFSIZE=4194304
export LMCACHE_USE_EXPERIMENTAL=True
export LMCACHE_LOCAL_CPU=True
export LMCACHE_MAX_LOCAL_CPU_SIZE=16.0
export LMCACHE_CUFILE_BUFFER_SIZE="4096"
export LMCACHE_GDS_PATH="$KV_DIR"
export LMCACHE_MAX_LOCAL_DISK_SIZE="$MAX_GB"
echo "=== [4/4] Starting vLLM TP=4 Distributed Server over Ray ($HEAD_IP:6379) ==="
exec vllm serve "$MODEL" \
--tensor-parallel-size 4 \
--distributed-executor-backend ray \
--quantization nvfp4 \
--max-model-len 131072 \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' \
--host 0.0.0.0 --port 8000