TinkerLab
ARTIFACT // RUN-EDGE-16GB-NPU-OR-BITNET-CPU.SH (SH)

16GB Edge NPU (Axelera axllm / Mobilint vllm-mblt) & BitNet b1.58 (bitnet.cpp I2_S LUT) Runner (run-edge-16gb-npu-or-bitnet-cpu.sh)

Audits PCIe/M.2 link state and executes 7B–8B models on Axelera Metis M.2 Max 16GB (Voyager SDK axllm) or Mobilint ARIES MLA100 16GB (vllm-mblt), or runs Microsoft BitNet-b1.58-2B-4T via ternary I2_S LUT kernels on pure CPU.

[KEY VERIFIED TAKEAWAY / TL;DR]This reproducible configuration artifact "16GB Edge NPU (Axelera axllm / Mobilint vllm-mblt) & BitNet b1.58 (bitnet.cpp I2_S LUT) Runner" (run-edge-16gb-npu-or-bitnet-cpu.sh, SH) is paired with Tinker AI Lab's hardware-verified guide "axelera-metis-m2-max-16gb-kinara-ara2-mobilint-aries-mla100-edge-npu-and-bitnet-b158-tmac-cpu-llm-benchmark". It exposes 4 customizable browser variables (EDGE_BACKEND=bitnet_cpu, BITNET_MODEL_PATH=models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf, NPU_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct, CPU_PHYSICAL_THREADS=4) with loopback-first security defaults.
01 // HOMELABLocal AI / HomeLab· Verified: 2026-11-13
Browse All 60 Artifacts →
run-edge-16gb-npu-or-bitnet-cpu.sh

16GB Edge NPU (Axelera axllm / Mobilint vllm-mblt) & BitNet b1.58 (bitnet.cpp I2_S LUT) Runner

Audits PCIe/M.2 link state and executes 7B–8B models on Axelera Metis M.2 Max 16GB (Voyager SDK axllm) or Mobilint ARIES MLA100 16GB (vllm-mblt), or runs Microsoft BitNet-b1.58-2B-4T via ternary I2_S LUT kernels on pure CPU.

View Paired Guide →
Customizable Variables (Saved in Browser)
UTF-8 Encoded · Live Client-Side Compilation
#!/usr/bin/env bash
# Tinker AI — 16GB Edge NPUs (Axelera Metis M.2 Max / Mobilint MLA100) & BitNet b1.58 Runner (Guide #50)
set -euo pipefail

BACKEND="bitnet_cpu"
BITNET_GGUF="models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf"
NPU_MODEL="meta-llama/Llama-3.1-8B-Instruct"
THREADS="4"

if [ "$BACKEND" = "bitnet_cpu" ]; then
  echo "=== [BitNet b1.58 Mode] Running Native 1.58-Bit Ternary ({-1,0,+1}) I2_S LUT Kernel on $THREADS Cores ==="
  lscpu | grep -E "Model name|Architecture|CPU\(s\)" || true
  ./build/bin/llama-cli \
    -m "$BITNET_GGUF" \
    -t "$THREADS" \
    -c 4096 \
    -n 256 \
    --temp 0.3 \
    -p "Explain why ternary {-1, 0, +1} weights replace floating-point FMA with integer LUT additions:"
elif [ "$BACKEND" = "axelera_m2" ]; then
  echo "=== [Axelera Metis M.2 Max 16GB Mode] Checking PCIe M.2 Device & Running Voyager SDK axllm ==="
  lspci | grep -i "Axelera" || true
  axdownloadmodel "$NPU_MODEL" || true
  axllm run "$NPU_MODEL" --ui --port 7860
elif [ "$BACKEND" = "mobilint_pcie" ]; then
  echo "=== [Mobilint ARIES MLA100 16GB 25W Mode] Checking NPU & Starting vllm-mblt Server ==="
  lspci | grep -i "Mobilint" || true
  vllm serve "$NPU_MODEL" \
    --device mblt \
    --max-model-len 8192 \
    --host 0.0.0.0 --port 8000
else
  echo "Unknown EDGE_BACKEND=$BACKEND (expected: bitnet_cpu | axelera_m2 | mobilint_pcie)" >&2
  exit 1
fi
PARAMETER SPECIFICATION // 参数化变量规范说明表

Customizable Parameter Reference for run-edge-16gb-npu-or-bitnet-cpu.sh

This SH artifact exposes 4 template variables compiled client-side in your browser.

Template TokenParameter DescriptionVerified DefaultExample Format
{{EDGE_BACKEND}}Edge Execution Target (bitnet_cpu | axelera_m2 | mobilint_pcie)bitnet_cpubitnet_cpu
{{BITNET_MODEL_PATH}}BitNet b1.58 GGUF (I2_S Ternary Kernel)models/BitNet-b1.58-2B-4T/ggml-model-i2_s.ggufmodels/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
{{NPU_MODEL_NAME}}16GB Edge NPU 8B Model Slug (axllm / vllm-mblt)meta-llama/Llama-3.1-8B-Instructmeta-llama/Llama-3.1-8B-Instruct
{{CPU_PHYSICAL_THREADS}}Physical CPU Threads for BitNet I2_S LUT (-t)44
PAIRED HARDWARE MANIFEST // 配套实测环境与边界记录

Verified Environment for "16GB Edge GenAI NPUs & Native 1.58-Bit Ternary LLMs: Axelera Metis M.2 Max (16GB · 6.5W), Kinara Ara-2 & Mobilint ARIES MLA100 (25W · vllm-mblt) vs Microsoft BitNet b1.58 2B4T (bitnet.cpp & T-MAC LUT Kernels)"

Tested Hardware & OS

Edge NPUs: Axelera Metis M.2 Max (16GB dedicated RAM · 214 TOPS DIMC · ~6.5W M.2 2280), Kinara Ara-2 Ara240 (16GB LPDDR4X M.2 · ~10W), Mobilint ARIES MLA100 (16GB LPDDR5 low-profile PCIe · 80 TOPS · 25W) | Pure CPU BitNet Hosts: Raspberry Pi 5 (8GB Cortex-A76), Snapdragon X Elite / Apple M4 ARM64, AMD Ryzen 7 8840U AVX2

Pinned Stack & Model Versions
Axelera Voyager SDK 1.4+ (axllm / axdownloadmodel) & Mobilint qb SDK + vllm-mblt plugin (vllm serve --device mblt)Microsoft bitnet.cpp (1.58-bit I2_S & TL1 / TL2 T-MAC Lookup-Table kernels)Models: microsoft/bitnet-b1.58-2B-4T-gguf (ggml-model-i2_s.gguf ~1.19 GB, non-embedding weights ~0.45 GB), Llama-3.1-8B-Instruct & Qwen2.5-7B-Instruct (W4A8 / INT4 on 16GB Edge NPUs)
Offline Boundary (100% Air-Gapped Edge NPU & Pure-CPU Ternary Execution)

All axllm, vllm-mblt, and bitnet.cpp (I2_S / T-MAC) binaries execute 100% locally without cloud APIs or telemetry.

Known Operational Limits & Failure Modes
  • Vision-Only M.2 NPU Trap (DEEPX DX-M1 / Hailo-8 / Mobilint REGULUS Cannot Run 7B LLMs): Many buyers confuse 20–40 TOPS vision M.2 cards (like DEEPX DX-M1 or 3W Mobilint REGULUS) with generative LLM accelerators. DX-M1 accelerates CNNs and ViT encoders at 2W–5W, but lacks onboard multi-gigabyte DRAM and autoregressive transformer decoder support—for 7B–8B local LLMs, you must choose a 16GB-equipped NPU (Axelera Metis M.2 Max 16GB, Kinara Ara-2 16GB, or Mobilint ARIES MLA100 16GB) or 8GB Hailo-10H (for 1.5B–3B).
  • BitNet b1.58 Requires Native Ternary QAT Checkpoints (Cannot Convert Standard FP16 Models via PTQ): You cannot take a standard Qwen3-8B or Llama-3.1-8B checkpoint and quantize it to 1.58-bit after training without destroying output quality. You must use models trained from scratch with Quantization-Aware Training (QAT) in ternary {-1, 0, +1} space—such as microsoft/bitnet-b1.58-2B-4T (trained on 4 Trillion tokens)—and compile bitnet.cpp using setup_env.py -md ... -q i2_s.
  • bitnet.cpp Submodule & Activation Gotcha (ReLU² vs SiLU): Official bitnet-b1.58-2B-4T uses squared ReLU (relu2) in its feed-forward layers and W1.58A8 activations. Running a bitnet-b1.58 GGUF inside an outdated generic llama.cpp binary that lacks the I2_S / relu2 path will either fail to load or output gibberish.
DEPLOYMENT CHECKLIST // 配置落地核验步骤

3-Step Verification Checklist After Deploying run-edge-16gb-npu-or-bitnet-cpu.sh

  1. Confirmed M.2/PCIe edge NPU has dedicated onboard DRAM (16GB on Axelera Metis M.2 Max / Kinara Ara-2 / Mobilint MLA100, or 8GB on Hailo-10H) rather than a vision-only CNN accelerator (DEEPX DX-M1 / Hailo-8).
  2. Compiled bitnet.cpp with Clang >= 18 via python3 setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s and set -t to physical CPU cores.
  3. Verified M.2 / PCIe link speed via lspci -vv and disabled aggressive PCIe ASPM power sleep (pcie_aspm=off) if NPU DMA transfers timeout under sustained decode.