16GB Edge NPU (Axelera axllm / Mobilint vllm-mblt) & BitNet b1.58 (bitnet.cpp I2_S LUT) Runner (run-edge-16gb-npu-or-bitnet-cpu.sh)
Audits PCIe/M.2 link state and executes 7B–8B models on Axelera Metis M.2 Max 16GB (Voyager SDK axllm) or Mobilint ARIES MLA100 16GB (vllm-mblt), or runs Microsoft BitNet-b1.58-2B-4T via ternary I2_S LUT kernels on pure CPU.
16GB Edge NPU (Axelera axllm / Mobilint vllm-mblt) & BitNet b1.58 (bitnet.cpp I2_S LUT) Runner
Audits PCIe/M.2 link state and executes 7B–8B models on Axelera Metis M.2 Max 16GB (Voyager SDK axllm) or Mobilint ARIES MLA100 16GB (vllm-mblt), or runs Microsoft BitNet-b1.58-2B-4T via ternary I2_S LUT kernels on pure CPU.
#!/usr/bin/env bash
# Tinker AI — 16GB Edge NPUs (Axelera Metis M.2 Max / Mobilint MLA100) & BitNet b1.58 Runner (Guide #50)
set -euo pipefail
BACKEND="bitnet_cpu"
BITNET_GGUF="models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf"
NPU_MODEL="meta-llama/Llama-3.1-8B-Instruct"
THREADS="4"
if [ "$BACKEND" = "bitnet_cpu" ]; then
echo "=== [BitNet b1.58 Mode] Running Native 1.58-Bit Ternary ({-1,0,+1}) I2_S LUT Kernel on $THREADS Cores ==="
lscpu | grep -E "Model name|Architecture|CPU\(s\)" || true
./build/bin/llama-cli \
-m "$BITNET_GGUF" \
-t "$THREADS" \
-c 4096 \
-n 256 \
--temp 0.3 \
-p "Explain why ternary {-1, 0, +1} weights replace floating-point FMA with integer LUT additions:"
elif [ "$BACKEND" = "axelera_m2" ]; then
echo "=== [Axelera Metis M.2 Max 16GB Mode] Checking PCIe M.2 Device & Running Voyager SDK axllm ==="
lspci | grep -i "Axelera" || true
axdownloadmodel "$NPU_MODEL" || true
axllm run "$NPU_MODEL" --ui --port 7860
elif [ "$BACKEND" = "mobilint_pcie" ]; then
echo "=== [Mobilint ARIES MLA100 16GB 25W Mode] Checking NPU & Starting vllm-mblt Server ==="
lspci | grep -i "Mobilint" || true
vllm serve "$NPU_MODEL" \
--device mblt \
--max-model-len 8192 \
--host 0.0.0.0 --port 8000
else
echo "Unknown EDGE_BACKEND=$BACKEND (expected: bitnet_cpu | axelera_m2 | mobilint_pcie)" >&2
exit 1
fi
Customizable Parameter Reference for run-edge-16gb-npu-or-bitnet-cpu.sh
This SH artifact exposes 4 template variables compiled client-side in your browser.
| Template Token | Parameter Description | Verified Default | Example Format |
|---|---|---|---|
{{EDGE_BACKEND}} | Edge Execution Target (bitnet_cpu | axelera_m2 | mobilint_pcie) | bitnet_cpu | bitnet_cpu |
{{BITNET_MODEL_PATH}} | BitNet b1.58 GGUF (I2_S Ternary Kernel) | models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf | models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf |
{{NPU_MODEL_NAME}} | 16GB Edge NPU 8B Model Slug (axllm / vllm-mblt) | meta-llama/Llama-3.1-8B-Instruct | meta-llama/Llama-3.1-8B-Instruct |
{{CPU_PHYSICAL_THREADS}} | Physical CPU Threads for BitNet I2_S LUT (-t) | 4 | 4 |
Verified Environment for "16GB Edge GenAI NPUs & Native 1.58-Bit Ternary LLMs: Axelera Metis M.2 Max (16GB · 6.5W), Kinara Ara-2 & Mobilint ARIES MLA100 (25W · vllm-mblt) vs Microsoft BitNet b1.58 2B4T (bitnet.cpp & T-MAC LUT Kernels)"
Edge NPUs: Axelera Metis M.2 Max (16GB dedicated RAM · 214 TOPS DIMC · ~6.5W M.2 2280), Kinara Ara-2 Ara240 (16GB LPDDR4X M.2 · ~10W), Mobilint ARIES MLA100 (16GB LPDDR5 low-profile PCIe · 80 TOPS · 25W) | Pure CPU BitNet Hosts: Raspberry Pi 5 (8GB Cortex-A76), Snapdragon X Elite / Apple M4 ARM64, AMD Ryzen 7 8840U AVX2
axllm / axdownloadmodel) & Mobilint qb SDK + vllm-mblt plugin (vllm serve --device mblt)Microsoft bitnet.cpp (1.58-bit I2_S & TL1 / TL2 T-MAC Lookup-Table kernels)Models: microsoft/bitnet-b1.58-2B-4T-gguf (ggml-model-i2_s.gguf ~1.19 GB, non-embedding weights ~0.45 GB), Llama-3.1-8B-Instruct & Qwen2.5-7B-Instruct (W4A8 / INT4 on 16GB Edge NPUs)All axllm, vllm-mblt, and bitnet.cpp (I2_S / T-MAC) binaries execute 100% locally without cloud APIs or telemetry.
- Vision-Only M.2 NPU Trap (
DEEPX DX-M1/Hailo-8/Mobilint REGULUSCannot Run 7B LLMs): Many buyers confuse 20–40 TOPS vision M.2 cards (likeDEEPX DX-M1or 3WMobilint REGULUS) with generative LLM accelerators.DX-M1accelerates CNNs and ViT encoders at 2W–5W, but lacks onboard multi-gigabyte DRAM and autoregressive transformer decoder support—for7B–8Blocal LLMs, you must choose a 16GB-equipped NPU (Axelera Metis M.2 Max 16GB,Kinara Ara-2 16GB, orMobilint ARIES MLA100 16GB) or8GB Hailo-10H(for1.5B–3B). BitNet b1.58Requires Native Ternary QAT Checkpoints (Cannot Convert Standard FP16 Models via PTQ): You cannot take a standardQwen3-8BorLlama-3.1-8Bcheckpoint and quantize it to1.58-bitafter training without destroying output quality. You must use models trained from scratch with Quantization-Aware Training (QAT) in ternary{-1, 0, +1}space—such asmicrosoft/bitnet-b1.58-2B-4T(trained on 4 Trillion tokens)—and compilebitnet.cppusingsetup_env.py -md ... -q i2_s.bitnet.cppSubmodule & Activation Gotcha (ReLU²vsSiLU): Officialbitnet-b1.58-2B-4Tuses squared ReLU (relu2) in its feed-forward layers andW1.58A8activations. Running abitnet-b1.58GGUF inside an outdated genericllama.cppbinary that lacks theI2_S/relu2path will either fail to load or output gibberish.
3-Step Verification Checklist After Deploying run-edge-16gb-npu-or-bitnet-cpu.sh
- Confirmed M.2/PCIe edge NPU has dedicated onboard DRAM (
16GBon Axelera Metis M.2 Max / Kinara Ara-2 / Mobilint MLA100, or8GBon Hailo-10H) rather than a vision-only CNN accelerator (DEEPX DX-M1/Hailo-8). - Compiled
bitnet.cppwithClang >= 18viapython3 setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_sand set-tto physical CPU cores. - Verified M.2 / PCIe link speed via
lspci -vvand disabled aggressive PCIe ASPM power sleep (pcie_aspm=off) if NPU DMA transfers timeout under sustained decode.