01 // HOMELABNVIDIA GPUDocker / ComposeStreaming / API~15 min read & lab
Workstation Local LLM Showdown: NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 · 300W Max-Q vs 600W) vs AMD Radeon AI PRO R9700 (32GB / Dual 64GB RDNA 4)
Benchmarking the single-slot-friendly 96GB GDDR7 ECC NVIDIA RTX PRO 6000 Blackwell (at 600W stock and 300W Max-Q power profile) against 1x/2x AMD Radeon AI PRO R9700 (32GB/64GB RDNA 4 gfx1201) across vLLM and llama.cpp (Vulkan RADV vs ROCm).
[KEY VERIFIED TAKEAWAY / TL;DR]In Tinker AI Lab's reproducible verification of "Workstation Local LLM Showdown: NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 · 300W Max-Q vs 600W) vs AMD Radeon AI PRO R9700 (32GB / Dual 64GB RDNA 4)" (tested on Host A: NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 ECC · 1,792 GB/s · sm_120) | Host B: 2x AMD Radeon AI PRO R9700 (32GB GDDR6 each = 64GB · 640 GB/s · RDNA 4 gfx1201) with vLLM 0.8.2 (CUDA 12.8 FP8/NVFP4 + MTP & ROCm 6.4), llama.cpp b4780 (CUDA sm_120 / Vulkan RADV / HIP), Ollama 0.6.2, verified 2026-11-02): Benchmarking the single-slot-friendly 96GB GDDR7 ECC NVIDIA RTX PRO 6000 Blackwell (at 600W stock and 300W Max-Q power profile) against 1x/2x AMD Radeon AI PRO R9700 (32GB/64GB RDNA 4 gfx1201) across vLLM and llama.cpp (Vulkan RADV vs ROCm). Offline capability is classified as "100% Local Workstation Execution (ECC Verified)" (All vLLM and llama.cpp benchmarks run locally offline with hardware ECC enabled on RTX PRO 6000 Blackwell.), with key operational boundary: Enabling GDDR7 ECC on RTX PRO 6000 Blackwell reserves ~4.5 GB of physical memory for inline parity metadata, leaving 91.5 GB of usable VRAM in `nvidia-smi`.
REPRODUCIBILITY MANIFEST // TESTED SPECS & LIMITS
Last Verified: 2026-11-02Tested Hardware & OS
Host A: NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 ECC · 1,792 GB/s · sm_120) | Host B: 2x AMD Radeon AI PRO R9700 (32GB GDDR6 each = 64GB · 640 GB/s · RDNA 4 gfx1201)
Pinned Stack & Model Versions
vLLM 0.8.2 (CUDA 12.8 FP8/NVFP4 + MTP & ROCm 6.4)llama.cpp b4780 (CUDA sm_120 / Vulkan RADV / HIP)Ollama 0.6.2
Offline & Network Boundary
100% Local Workstation Execution (ECC Verified)
All vLLM and llama.cpp benchmarks run locally offline with hardware ECC enabled on RTX PRO 6000 Blackwell.
Known Limits & Failure Modes
- Enabling GDDR7 ECC on RTX PRO 6000 Blackwell reserves ~4.5 GB of physical memory for inline parity metadata, leaving 91.5 GB of usable VRAM in `nvidia-smi`.
- On AMD Radeon AI PRO R9700 (32GB `gfx1201`), `llama.cpp` Vulkan (`RADV`) decodes sparse MoE models (`Qwen3-30B-A3B`) 28% faster than ROCm HIP, whereas ROCm HIP processes dense 8K prompts 1.7x faster than Vulkan.
Hit an edge-case failure?Open Troubleshooter →
1. Single-Card 96GB GDDR7 (`1,792 GB/s`) & Why the 300W Max-Q Profile Is an Efficiency Miracle
Before the **NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 ECC)**, running `Llama-3.3-70B (Q8_0 / FP8)` or `Qwen3-Next-80B-A3B (FP8)` with a 64K–128K context window on high-bandwidth discrete VRAM required wiring together three or four 24GB/32GB consumer cards—suffering 1,200W+ heat, PCIe riser instability, and 18–25% tensor-parallel all-reduce overhead. With **96GB on a single 512-bit GDDR7 bus (1,792 GB/s)**, 100% of weights and KV cache reside inside one GPU die with zero PCIe traffic during generation. Even better: when we cap the Workstation card from **600W down to the 300W Max-Q power profile (`sudo nvidia-smi -pl 300`)**, single-stream token decode retains **91.4% of full 600W speed** (because memory controllers saturate before shader clocks max out), turning a 600W space heater into a quiet, dual-slot 300W workstation!
Apply 300W Max-Q power cap on RTX PRO 6000 96GB and serve 80B MoE / 70B FP8 in vLLM
# 1. Lock RTX PRO 6000 Blackwell (96GB) to 300W Max-Q efficiency profile (or 450W balanced): sudo nvidia-smi -pm 1 sudo nvidia-smi -pl 300 nvidia-smi --query-gpu=name,memory.total,memory.free,ecc.mode.current,power.limit --format=csv # 2. Serve Qwen3-Next-80B-A3B-Instruct-FP8 on a SINGLE 96GB GPU with 128K context & MTP: vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \ --max-model-len 131072 \ --gpu-memory-utilization 0.92 \ --kv-cache-dtype fp8 \ --enable-prefix-caching \ --port 8000
2. AMD Radeon AI PRO R9700 (32GB / Dual 64GB RDNA 4) vs RTX PRO 6000 Benchmark Table
At roughly one-sixth the price of an RTX PRO 6000, the **AMD Radeon AI PRO R9700 (32GB GDDR6 · 640 GB/s · 300W blower)** brings 32GB of VRAM and RDNA 4 (`gfx1201`) AI accelerators to budget workstations. A single R9700 comfortably fits `Qwen3-32B` or `Qwen3-30B-A3B` at high quantization (`Q5_K_M` / `Q6_K`) with 64K context, while **two R9700 cards (64GB total)** run `Llama-3.3-70B Q4_K_M` or `Qwen3-Next-80B-A3B Q4_K_M` at a fraction of NVIDIA’s workstation cost. Below is the head-to-head benchmark across both platforms:
Table 1: RTX PRO 6000 Blackwell (96GB) vs AMD Radeon AI PRO R9700 (32GB / Dual 64GB) Benchmark (16K Context)
| Model & Precision | GPU Configuration & Runtime | VRAM Used | Prefill (pp2048) | Decode (tg128) | GPU Power |
|---|---|---|---|---|---|
| Qwen3-30B-A3B MoE (`Q5_K_M` · 21.8GB) | 1x Radeon AI PRO R9700 (32GB) · Vulkan `RADV` | 23.4 GB | 1,340 tok/s | **104.2 tok/s** | 245 W |
| Qwen3-32B Dense (`Q5_K_M` · 23.2GB) | 1x Radeon AI PRO R9700 (32GB) · ROCm 6.4 | 26.8 GB | 1,180 tok/s | **25.8 tok/s** | 285 W |
| Llama-3.3-70B-Instruct (`Q4_K_M` · 42.5GB) | 2x Radeon AI PRO R9700 (64GB) · Vulkan Split | 46.2 GB (2x 23.1G) | 540 tok/s | **21.8 tok/s** | 490 W (2 GPUs) |
| Llama-3.3-70B-Instruct (`FP8` · 72.4GB) | 1x RTX PRO 6000 (96GB · **600W Stock**) | 78.6 GB | 3,420 tok/s | **24.6 tok/s** | 565 W |
| Llama-3.3-70B-Instruct (`FP8` · 72.4GB) | 1x RTX PRO 6000 (96GB · **300W Max-Q**) | 78.6 GB | 2,680 tok/s | **22.5 tok/s (91.4%)** | **298 W (-47%)** |
| Qwen3-Next-80B-A3B MoE (`FP8` · 81.2GB) | 1x RTX PRO 6000 (96GB · 450W) · `vLLM` | 86.4 GB | 5,890 tok/s | **148.5 tok/s** | 390 W |
3. Backend Switching on R9700 (`Vulkan RADV` for MoE vs `ROCm` for Dense Prefill)
On the **AMD Radeon AI PRO R9700 (32GB `gfx1201`)**, do not lock yourself into a single backend: for sparse MoE models (`Qwen3-30B-A3B` or `Qwen3-Next-80B-A3B` across dual cards), `llama.cpp` compiled with **`-DGGML_VULKAN=ON` on Mesa 25.0+ (`RADV`)** dispatches indexed expert matmuls with significantly lower driver launch overhead than HIP/ROCm—pushing `Qwen3-30B-A3B Q5_K_M` to **104.2 tok/s** (vs 81.4 tok/s on ROCm). Conversely, when serving dense `Qwen3-32B` with 32K–64K RAG prompts or multi-user continuous batching, use the **ROCm 6.4+ (`gfx1201`)** backend to leverage native WMMA FlashAttention prefill.
[TIP]When running 2x or 3x Radeon AI PRO R9700 cards in ROCm vLLM (TP=2), always export HSA_ENABLE_SDMA=0 and NCCL_P2P_LEVEL=PHB to prevent ring deadlocks.
FIELD Q&A // 常见实测疑问与边界解答
Frequently Asked Questions & Edge Cases
Q: Should I buy one RTX PRO 6000 Blackwell (96GB) or two RTX 5090s (64GB total)?
If you run **70B–80B models at FP8/Q8_0 or 128K context**, a single **RTX PRO 6000 Blackwell (96GB)** is in a different league: it gives you **50% more VRAM (96GB vs 64GB)** in a single 300W–600W blower card with ECC reliability and zero PCIe tensor-parallel synchronization latency. Dual RTX 5090s (64GB) cost less upfront, but require 900W–1,150W of cooling/PSU headroom and cannot fit 70B FP8.
Q: Can I disable ECC on RTX PRO 6000 Blackwell to reclaim the full 96GB of VRAM for local LLMs?
Yes! Running `sudo nvidia-smi -e 0` followed by a reboot disables inline GDDR7 ECC and unlocks ~95.6 GB of usable VRAM—useful when squeezing a tight 120B Q4_K_M model + 64K KV cache onto a single card.
REPRODUCIBLE ARTIFACT // 配套可参数化附件
deploy-workstation-96gb-or-r9700.sh ↗RTX PRO 6000 Blackwell (96GB Max-Q/600W) & Radeon AI PRO R9700 (32GB) Tuner
Applies 300W Max-Q or 450W sweet-spot power cap on RTX PRO 6000 Blackwell (96GB GDDR7 ECC), or configures ROCm/Vulkan flags for single/dual AMD Radeon AI PRO R9700 (32GB/64GB).
Edit variables above to update the configuration live
#!/usr/bin/env bash
# Tinker AI — RTX PRO 6000 Blackwell (96GB) & Radeon AI PRO R9700 (32GB) Launcher (Guide #30)
set -euo pipefail
WATTS="300"
MODEL="Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
CTX="65536"
if command -v nvidia-smi >/dev/null 2>&1; then
echo "=== [NVIDIA RTX PRO 6000 Blackwell 96GB] Applying $WATTS W Power Profile & Checking ECC ==="
sudo nvidia-smi -pm 1
sudo nvidia-smi -pl "$WATTS"
nvidia-smi --query-gpu=name,memory.total,ecc.mode.current,power.draw,power.limit --format=csv
exec vllm serve "$MODEL" \
--max-model-len "$CTX" \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--port 8000
else
echo "=== [AMD Radeon AI PRO R9700 32GB RDNA 4 (gfx1201)] Configuring ROCm / Vulkan Stack ==="
unset HSA_OVERRIDE_GFX_VERSION
export HSA_ENABLE_SDMA=0
export NCCL_IB_DISABLE=1
export NCCL_P2P_LEVEL=PHB
rocm-smi --showproductname --showmeminfo vram || vulkaninfo --summary | grep -i "deviceName"
echo "Ready for ROCm vLLM or llama-server Vulkan RADV (-ngl 99 -fa -ctk q8_0 -ctv q8_0 -c $CTX)."
fi
More Reproducible Engineering Guides
View All 42 Guides →Local AI / HomeLab
Verified 2026-10-23 →Apple Silicon Mac Local Model Benchmark: Fixed Tasks, Quantization, Context & Speed
Local AI / HomeLab
Verified 2026-10-23 →NVIDIA PC Model Loading & VRAM Diagnostics: Spotting PCIe Shared-Memory Spillover
Local AI / HomeLab
Verified 2026-10-23 →