Local Model Slowdown, OOM, or Silent CPU Fallback

Open WebUI or client tools often request 16K–32K context. At FP16, the KV cache adds 3–8 GB on top of the model weights, forcing layers onto slow system RAM.

1. Enable Flash Attention and Q8_0 KV Cache quantization on the Ollama server (cuts KV memory by ~50%).
2. Cap num_ctx to 8192 or 16384 in your Modelfile or client request options.

# Set in environment before starting Ollama
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_NUM_PARALLEL=1

[BOUNDARY]Use the Tinker VRAM Calculator (/tools/vram-calculator/) to preview exact weight + KV cache budgets.

When an embedding model (e.g., nomic-embed-text) and a chat LLM are called together in RAG, or when switching models, previous models remain resident for 5 minutes.

1. Stop resident models immediately with ollama stop <model>.
2. Set OLLAMA_MAX_LOADED_MODELS=1 (or 2 if running tiny embedding model + LLM).

ollama stop $(ollama ps | awk 'NR>1 {print $1}')
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_KEEP_ALIVE=5m

[BOUNDARY]If using Local RAG, keep your embedding model lightweight (e.g. bge-m3 or nomic-embed-text) so both fit.

Docker containers cannot see host NVIDIA GPUs unless nvidia-container-toolkit is installed and deploy.resources.reservations.devices is declared.

1. Install nvidia-container-toolkit on the host and run sudo nvidia-ctk runtime configure --runtime=docker.
2. Restart Docker daemon and add the GPU reservation block in docker-compose.yml.

services:
  ollama:
    image: ollama/ollama:0.5.7
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

[BOUNDARY]Note: Docker Desktop on macOS cannot pass Apple Silicon Metal GPU into Linux containers; run Ollama natively on macOS host instead.