04 // DIAGNOSISDocker / ComposeNVIDIA GPUApple Silicon MacStreaming / API~9 min read & lab

Troubleshooting RAG Embedding Dimension Mismatch (1024 vs 768) & Zombie VRAM Allocation

Fix the two most common post-setup local AI failures: Chroma/PGVector dimension mismatch errors after changing embedding models, and GPU VRAM staying 95% full when ollama ps shows zero models.

[KEY VERIFIED TAKEAWAY / TL;DR]In Tinker AI Lab's reproducible verification of "Troubleshooting RAG Embedding Dimension Mismatch (1024 vs 768) & Zombie VRAM Allocation" (tested on Linux / macOS / Windows WSL2 · Ollama 0.5.7 · Open WebUI / Dify / Continue (ChromaDB, Qdrant, PGVector) with Ollama 0.5.7, bge-m3 (1024-dim), nomic-embed-text (768-dim), mxbai-embed-large (1024-dim), verified 2026-10-26): Fix the two most common post-setup local AI failures: Chroma/PGVector dimension mismatch errors after changing embedding models, and GPU VRAM staying 95% full when ollama ps shows zero models. Offline capability is classified as "100% Local Vector DB & GPU Process Diagnostics" (All vector dimension probes and GPU process checks run locally via curl and nvidia-smi.), with key operational boundary: Vector spaces are mathematically incompatible across models: even if two models share the same dimension (`bge-m3` is 1024-dim and `mxbai-embed-large` is 1024-dim), querying an index built with model A using model B returns pure random noise!

REPRODUCIBILITY MANIFEST // TESTED SPECS & LIMITS

Last Verified: 2026-10-26

Tested Hardware & OS

Linux / macOS / Windows WSL2 · Ollama 0.5.7 · Open WebUI / Dify / Continue (ChromaDB, Qdrant, PGVector)

Pinned Stack & Model Versions

Ollama 0.5.7bge-m3 (1024-dim)nomic-embed-text (768-dim)mxbai-embed-large (1024-dim)

Offline & Network Boundary

100% Local Vector DB & GPU Process Diagnostics

All vector dimension probes and GPU process checks run locally via curl and nvidia-smi.

Known Limits & Failure Modes

Vector spaces are mathematically incompatible across models: even if two models share the same dimension (`bge-m3` is 1024-dim and `mxbai-embed-large` is 1024-dim), querying an index built with model A using model B returns pure random noise!
In WSL2 on Windows, freeing CUDA memory inside Linux does not always immediately shrink `vmmemWSL` on Windows Task Manager without `OLLAMA_KEEP_ALIVE=0` or `wsl --shutdown`.

Hit an edge-case failure?Open Troubleshooter →

1. Local Embedding Model Dimension & Context Matrix (`768` vs `1024` vs `384`)

When you create a knowledge collection in ChromaDB, Qdrant, or PostgreSQL `pgvector`, the table column is permanently locked to the vector length of the first embedded chunk (e.g. `vector(768)` or `vector(1024)`). If you later switch your RAG setting from default `all-MiniLM-L6-v2` (`384`) or `nomic-embed-text` (`768`) to `bge-m3` (`1024`), every insert or query throws `InvalidDimensionException` or HTTP 500:

Table 1: Verified Local Embedding Models — Vector Dimensions, Max Tokens & Compatibility

Embedding Model	Vector Dimension	Max Context Window	Best Use Case & Caveat
all-MiniLM-L6-v2 (Built-in default)	384-dim	256 tokens (Very short!)	Weak on Chinese/CJK; switch to bge-m3 + re-index
nomic-embed-text	768-dim	8192 tokens	Fast English code/docs; incompatible with 1024-dim tables
bge-m3 (Recommended Bilingual)	1024-dim	8192 tokens	Top CJK + English recall; requires ~1.2GB VRAM
mxbai-embed-large	1024-dim	512 tokens	Same 1024-dim as bge-m3, but DIFFERENT vector space!

Verify Exact Embedding Vector Dimension & Flush Idle VRAM in 10 Seconds

# 1. Probe exact vector dimension returned by your local Ollama embedding model
curl -s http://127.0.0.1:11434/api/embeddings \
  -d '{"model": "bge-m3", "prompt": "dimension check"}' | jq '.embedding | length'
# Expected output for bge-m3: 1024 (nomic-embed-text: 768)

# 2. Immediately unload a model from VRAM without restarting the Ollama service
curl -s http://127.0.0.1:11434/api/generate -d '{"model": "qwen2.5:14b", "keep_alive": 0}'

2. Diagnosing "Zombie VRAM": Why GPU Memory Stays Full When `ollama ps` Is Empty

If `ollama ps` reports zero loaded models yet your 16GB/24GB GPU shows 14GB in use, check three culprits via `nvidia-smi`: 1. **Built-in Whisper / Sentence-Transformers inside Docker**: Open WebUI or Dify running an in-container Python `uvicorn` process that loaded a local Whisper or HuggingFace Reranker model onto CUDA. 2. **Orphaned `ollama_llama_server` Runner**: A crashed generation left a detached runner process alive (`pkill -9 -f ollama_llama_server`). 3. **Second Serving Engine**: A background `vllm`, `llama-server`, or ComfyUI instance pinning VRAM.

3. Coordinating Dual-Model VRAM (`Chat LLM` + `bge-m3` Embedding) on Single GPUs

In a RAG chat turn, your system embeds the user query with `bge-m3` (~1.2 GB) and immediately calls your chat LLM (`qwen2.5:14b`, ~10.2 GB). If `OLLAMA_MAX_LOADED_MODELS` is set to `1`, Ollama unloads the 14B model to load `bge-m3`, and then reloads the 14B model from disk 200ms later—adding 8–15 seconds of disk thrashing to every message! Set `OLLAMA_MAX_LOADED_MODELS=2` and budget an extra 1.2 GB in the VRAM Calculator.

FIELD Q&A // 常见实测疑问与边界解答

Frequently Asked Questions & Edge Cases

Q: How do I fix `InvalidDimensionException: Embedding dimension 1024 does not match collection dimension 384` in Open WebUI?

Open **Workspace -> Knowledge**, delete the old collections that were created when the 384-dim default model was active (or click **Reset Vector Storage** in Admin -> Settings -> Documents), and re-upload your documents so ChromaDB creates fresh 1024-dimensional tables.

Q: Why does Windows Task Manager still show `vmmemWSL` using 12GB of RAM even after I unloaded my Ollama model?

Linux inside WSL2 caches filesystem pages (including `.gguf` model weights) in RAM and only returns freed memory to Windows gradually. Add `autoMemoryReclaim=gradual` under `[experimental]` in `%USERPROFILE%\.wslconfig`, or run `echo 3 | sudo tee /proc/sys/vm/drop_caches` inside WSL2.

REPRODUCIBLE ARTIFACT // 配套可参数化附件

Local RAG Embedding Dimension, Bilingual Recall & Zombie VRAM Diagnostic Script

Probes exact output vector dimensions (e.g. 1024 for bge-m3 vs 768 for nomic-embed-text), checks dual-model VRAM residency, and verifies non-empty embeddings.

check-rag-embedding-health.sh ↗

Embedding Model TagExpected Vector DB DimensionOllama Endpoint

Edit variables above to update the configuration live

#!/usr/bin/env bash
# Tinker AI — RAG Embedding Dimension & Multi-Model VRAM Healthcheck (Guide #20)
set -euo pipefail

EMBED="bge-m3:latest"
EXP_DIM="1024"
API="http://127.0.0.1:11434/api/embed"

echo "=== [1/3] Probing Vector Dimension for $EMBED ==="
RESP=$(curl -sS "$API" -d "{
  \"model\": \"$EMBED\",
  \"input\": [\"Tinker AI Lab bilingual RAG verification\", \"本地知识库跨语言向量检索测试\"],
  \"keep_alive\": \"5m\"
}")

echo "$RESP" | python3 -c "
import sys, json, math
data = json.load(sys.stdin)
vecs = data.get('embeddings', [])
if not vecs:
    print('[FAIL] Empty embeddings array returned! Check model tag.')
    sys.exit(1)
dim = len(vecs[0])
exp = int('$EXP_DIM')
status = 'OK' if dim == exp else f'MISMATCH (Expected {exp}, Got {dim} -> Must re-index Vector DB!)'
dot = sum(a*b for a, b in zip(vecs[0], vecs[1]))
norm0 = math.sqrt(sum(a*a for a in vecs[0]))
norm1 = math.sqrt(sum(b*b for b in vecs[1]))
cos = dot / (norm0 * norm1) if (norm0 and norm1) else 0.0
print(f'Vector Dimension : {dim} [{status}]')
print(f'EN-ZH Cosine Sim : {cos:.4f} (Expect > 0.55 for multilingual models like bge-m3)')
"

echo "=== [2/3] Inspecting Co-Loaded Models & VRAM Split ==="
ollama ps

echo "=== [3/3] Checking OLLAMA_MAX_LOADED_MODELS Policy ==="
echo "Tip: Ensure OLLAMA_MAX_LOADED_MODELS>=2 so $EMBED and your Chat LLM do not thrash disk on every RAG query."

VERIFICATION CHECKLIST

Reproducibility Checklist

0 / 3

Check off each verification milestone on your machine. Progress is saved locally in your browser.

Verified `jq ".embedding | length"` matches the vector collection dimension (`1024` for `bge-m3`, `768` for `nomic-embed-text`).Re-indexed existing knowledge collections after changing the embedding model (even if dimensions were equal).Configured `OLLAMA_MAX_LOADED_MODELS=2` so `bge-m3` and the main Chat LLM do not evict each other during RAG turns.

PAIRED DIAGNOSTIC TREE // 配套排障树