Hybrid CPU+GPU MoE (`NUMA` / `-ot` OOM) & Edge NPU (`Hailo-10H` / `RK3588 CMA`) Traps

On multi-CCD or dual-socket AMD EPYC / Intel Xeon servers with 256GB–512GB DDR5, Linux’s default `kernel.numa_balancing=1` constantly migrates 4KB memory pages of 350GB MoE expert weights across NUMA domains during token generation, saturating Infinity Fabric / UPI and cutting decode speed by 50–65%. Using SMT (hyperthreads) further stalls AVX-512 register pipelines.

1. Immediately disable kernel auto-NUMA page migration via `sudo sysctl -w kernel.numa_balancing=0` (and persist in `/etc/sysctl.d/99-llm-numa.conf`).
2. Launch `ik_llama.cpp` under `numactl --interleave=all` (or set `export USE_NUMA=1` in `KTransformers`) so 12-channel / 24-channel DDR5 controllers stream expert weights in parallel, and set `-t` strictly to physical core count.

# 1. Disable Linux Auto-NUMA page thrashing immediately:
sudo sysctl -w kernel.numa_balancing=0

# 2. Flush VFS page cache so 350GB MoE weights map cleanly into DDR5:
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

# 3. Launch ik_llama.cpp interleaved across all 12-ch DDR5 channels (physical cores only):
numactl --interleave=all ./build/bin/llama-server \
  -m /mnt/nvme/models/DeepSeek-R1-671B-IQ4_XS.gguf \
  -ngl 99 -ot "blk\.[0-9]+\.ffn_.*_exps\.weight=CPU" \
  -mla 3 -fa -fmoe -ctk q8_0 -ctv q8_0 -c 32768 -t 32

[BOUNDARY]Disabling `kernel.numa_balancing=0` + `numactl --interleave=all` raises DeepSeek-R1-671B (`IQ4_XS`) decode speed on 12-channel DDR5 EPYC from 6.1 tok/s to 15.8 tok/s.

When running `-ngl 99` on `DeepSeek-R1-671B` or `Qwen3-235B-A22B` with a single 24GB/32GB GPU, any typo in `-ot` (or forgetting to escape `\.` in bash) attempts to load 300GB+ of routed expert tensors (`ffn_gate_exps`, `ffn_up_exps`, `ffn_down_exps`) into GPU VRAM. On 24GB cards, keeping all 61 layers of shared experts + 32K FP16 KV cache also exceeds 24GB unless `-mla 3` and `q8_0` KV cache are enabled.

1. Use `-ot "blk\.[0-9]+\.ffn_.*_exps\.weight=CPU"` so all routed MoE expert matrices stay in system DDR5 RAM while shared layers and attention stay on CUDA.
2. On 24GB GPUs (RTX 3090 / 4090), enable `-mla 3 -fa -ctk q8_0 -ctv q8_0` (which compresses DeepSeek MLA KV cache by >70%), or offload layers 45–60 attention to CPU if running 64K+ context.

# Fit DeepSeek-R1-671B Attention + 32K KV Cache cleanly inside a single 24GB/32GB GPU:
./build/bin/llama-server \
  -m /mnt/nvme/models/DeepSeek-R1-671B-IQ4_XS.gguf \
  -ngl 99 \
  -ot "blk\.[0-9]+\.ffn_.*_exps\.weight=CPU" \
  -mla 3 -fa -fmoe \
  -ctk q8_0 -ctv q8_0 -c 32768 -t 32

[BOUNDARY]With `-ot "blk\.[0-9]+\.ffn_.*_exps\.weight=CPU"` and `-mla 3 -ctk q8_0`, DeepSeek-R1-671B occupies ~18.4 GB of GPU VRAM at 32K context—fitting safely on a single 24GB RTX 3090/4090 or 32GB RTX 5090 / PRO 4500.

The Raspberry Pi AI HAT+ 2 (`Hailo-10H` with 8GB onboard LPDDR4X) requires the GenAI-capable **HailoRT 5.x driver stack (`hailo-h10-all`)**. Having the legacy `hailo-all` metapackage (HailoRT 4.x for Hailo-8/8L vision chips) installed causes `HAILO_INVALID_FIRMWARE` or prevents `hailo-ollama` from accessing the 8GB NPU memory.

1. Purge `hailo-all` (`sudo apt purge -y hailo-all hailort`) and install `sudo apt install -y hailo-h10-all` followed by a cold reboot.
2. Add `dtparam=pciex1_gen=3` to `/boot/firmware/config.txt` so the Pi 5 PCIe x1 link runs at 8 GT/s (Gen 3) for fast prompt prefill and HEF model loading.

# 1. Switch from legacy Hailo-8 stack to Hailo-10H GenAI stack on Raspberry Pi 5:
sudo apt purge -y hailo-all hailort
sudo apt update && sudo apt install -y hailo-h10-all

# 2. Enable PCIe Gen 3.0 speed on Raspberry Pi 5:
grep -q "pciex1_gen=3" /boot/firmware/config.txt || echo "dtparam=pciex1_gen=3" | sudo tee -a /boot/firmware/config.txt
sudo reboot

# 3. Verify Hailo-10H firmware & launch hailo-ollama:
hailortcli fw-control identify
hailo-ollama run qwen2.5-instruct:1.5b

[BOUNDARY]Once `hailo-h10-all` is active, `Qwen2.5-1.5B` runs entirely inside the AI HAT+ 2’s 8GB onboard RAM at ~19.4 tok/s while Pi 5 host CPU load stays under 5%.

Rockchip’s 6 TOPS NPU driver (`rknpu` + `rkllm-runtime`) allocates model weights using physically contiguous DMA buffers (`DMA_BUF` / CMA) and locked pages. If `ulimit -l` is capped at 64KB or Linux VFS page fragmentation exhausts contiguous memory blocks, `rknn_init` fails with `-1` or `DMA_BUF_IOCTL_ALLOC failed` even when `free -h` shows 10GB+ free RAM.

1. Run `ulimit -l unlimited` and `echo 3 | sudo tee /proc/sys/vm/drop_caches && echo 1 | sudo tee /proc/sys/vm/compact_memory` before starting `rkllm_server`.
2. Lock the RK3588 NPU and DMC memory controller governors to `performance` (and set `cma=512M` in `/boot/armbianEnv.txt` or `extlinux.conf` if `CmaFree` is low).

# 1. Unlock memlock limit and defragment physical RAM for RK3588 NPU DMA:
ulimit -l unlimited
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
echo 1 | sudo tee /proc/sys/vm/compact_memory
grep -i "Cma" /proc/meminfo

# 2. Lock RK3588 6 TOPS NPU & DMC memory controller to maximum frequency:
echo performance | sudo tee /sys/class/devfreq/fdab0000.npu/governor 2>/dev/null || true
echo performance | sudo tee /sys/class/devfreq/dmc/governor 2>/dev/null || true

[BOUNDARY]Locking the RK3588 NPU + DMC governors to `performance` after memory compaction boosts `Qwen2.5-3B-W4A16.rkllm` speed from 8.4 tok/s to 12.6 tok/s.