Multi-GPU PCIe P2P Deadlocks (`iommu=pt` / ACS) & DGX Spark 200GbE RoCEv2 TCP Fallback

On a 2-node NVIDIA DGX Spark setup, Ray and NCCL default to the interface associated with the node hostname (often 1GbE LAN or Wi-Fi) unless explicitly pinned to the 200GbE QSFP112 interface (`enp1s0f0np0`) and Mellanox RDMA HCA (`mlx5_0:1` / `rocep1s0f0`).

1. Start `ray start --head --node-ip-address=192.168.10.1` using the dedicated `/30` subnet IP assigned to the 200GbE ConnectX-7 port (`enp1s0f0np0`).
2. Export `NCCL_SOCKET_IFNAME=enp1s0f0np0`, `NCCL_IB_HCA=mlx5_0:1`, and `NCCL_IB_DISABLE=0` on both nodes before launching `vllm serve`.

# Pin Ray & NCCL to the 200GbE ConnectX-7 QSFP112 interface:
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0:1
export NCCL_IB_GID_INDEX=3
export NCCL_SOCKET_IFNAME=enp1s0f0np0
export GLOO_SOCKET_IFNAME=enp1s0f0np0

# Verify ~197 Gb/s (24.6 GB/s) RDMA bandwidth before launching vLLM:
ib_write_bw -d rocep1s0f0 -F --report_gbits -s 1048576 -q 8 -b 192.168.10.1

[BOUNDARY]Confirm `NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE` appears in startup logs; 2-node Qwen3-235B-A22B throughput jumps from 2.8 tok/s (TCP) to 34.6 tok/s (RoCEv2).

RDMA Queue Pair (QP) handshake fails with `ibv_modify_qp` or `error 12` when one node defaults to RoCEv1 (GID index 0/1) while the other uses IPv4 RoCEv2 (GID index 3), or when one interface is left at `mtu 1500` while the peer sends 4096-byte RDMA payload frames.

1. Run `show_gids` on both nodes and locate the row where `VER` is `v2` and `IPv4` matches your `192.168.10.x` address (typically index `3`), then set `export NCCL_IB_GID_INDEX=3`.
2. Set `sudo ip link set dev enp1s0f0np0 mtu 9000` (and persist in `/etc/netplan/99-spark-cluster.yaml`) on both nodes.

# Set MTU 9000 jumbo frames on both ConnectX-7 endpoints:
sudo ip link set dev enp1s0f0np0 mtu 9000
ip -d link show enp1s0f0np0 | grep mtu

# Inspect RoCEv2 GID table and lock GID index 3:
show_gids | grep -E "192\.168\.10\.|v2"
export NCCL_IB_GID_INDEX=3

[BOUNDARY]On switchless back-to-back DAC links between two DGX Sparks, setting `mtu 9000` raises effective RDMA throughput from 154 Gb/s to 197.4 Gb/s.

On AMD Ryzen / Threadripper / EPYC and many consumer PCIe 5.0 motherboards, strict CPU IOMMU DMA remapping or PCIe Access Control Services (ACS) intercepts direct GPU-to-GPU BAR1 Peer-to-Peer transfers, causing `ncclAllReduce` to hang silently during `vLLM` Tensor Parallel startup.

1. For immediate recovery without rebooting, export `NCCL_P2P_DISABLE=1` (routes tensor all-reduce through host shared memory `SHM` with only ~8–12% overhead on PCIe 5.0 x8/x16).
2. For permanent full-speed P2P DMA, add `amd_iommu=on iommu=pt` (or `intel_iommu=on iommu=pt`) to `GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub`, disable ACS in BIOS if available, and reboot.

# Option A — Immediate workaround without reboot (uses PCIe SHM instead of BAR1 P2P):
export NCCL_P2P_DISABLE=1
export NCCL_SHM_DISABLE=0

# Option B — Permanent kernel fix for direct GPU-to-GPU PCIe P2P DMA:
sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt /' /etc/default/grub
sudo update-grub
# Reboot and verify P2P matrix:
nvidia-smi topo -p2p r

[BOUNDARY]Always verify with `nvidia-smi topo -p2p r`: if status shows `OK` after `iommu=pt`, remove `NCCL_P2P_DISABLE=1` to regain full 64 GB/s PCIe 5.0 P2P bandwidth.

When running 2x or 3x AMD Radeon AI PRO R9700 (32GB RDNA 4 `gfx1201`) cards under ROCm vLLM (`TP=2`), the asynchronous SDMA engine across PCIe Root Complex boundaries can trigger `MEMORY_APERTURE_VIOLATION` or RCCL ring deadlocks unless SDMA P2P is disabled or compute shaders handle copies.

1. For multi-GPU vLLM on ROCm (`vllm/vllm-openai-rocm`), export `HSA_ENABLE_SDMA=0`, `NCCL_IB_DISABLE=1`, and `NCCL_P2P_LEVEL=PHB` alongside `iommu=pt` on the host kernel.
2. For single-user 70B–120B GGUF inference across dual R9700 (64GB), use `llama-server -ngl 99 --split-mode layer` with the Vulkan (`RADV`) backend, which avoids RCCL all-reduce overhead entirely.

# Multi-card AMD Radeon AI PRO R9700 (gfx1201) RCCL stability env vars for vLLM:
export HSA_ENABLE_SDMA=0
export NCCL_IB_DISABLE=1
export NCCL_P2P_LEVEL=PHB
export NCCL_ALGO=Ring

# Or run dual-R9700 (64GB) GGUF inference via Vulkan RADV with zero RCCL overhead:
GGML_VK_VISIBLE_DEVICES=0,1 ./build/bin/llama-server \
  -m ./models/Llama-3.3-70B-Instruct-Q4_K_M.gguf -ngl 99 --split-mode layer -c 32768 -fa

[BOUNDARY]On dual R9700 (64GB total), `--split-mode layer` in `llama.cpp` Vulkan achieves 21.8 tok/s on Llama-3.3-70B Q4_K_M without requiring PCIe P2P BAR1 support.