LLM Streaming API Fails or Drops After HTTP 200 OK

Once an LLM server flushes HTTP 200 headers to start an SSE stream, it can no longer change the HTTP status code to 500 or 529 if an overload or OOM occurs mid-generation.

1. Check every parsed SSE JSON payload for `event.error` or `event.type === "error"` before appending delta text.
2. Discard partial corrupted output and trigger a bounded retry with exponential backoff.

const payload = JSON.parse(rawDataLine);
if (payload.error || payload.type === 'error') {
  throw new Error(`Mid-stream error after HTTP 200: ${payload.error?.message || JSON.stringify(payload)}`);
}

[BOUNDARY]Documented explicitly in Claude API & OpenAI-compatible streaming specifications.

Local prompt evaluation on 16K+ tokens can take 20–90 seconds before the first token is emitted. Nginx or gateway proxies default to 60s read timeout.

1. Set proxy_buffering off and proxy_read_timeout 300s in Nginx / reverse proxy.
2. Send periodic SSE keep-alive comments or set client chunk timeout >= 45s.

# Nginx location block for LLM SSE endpoints
location /v1/ {
    proxy_pass http://127.0.0.1:11434/v1/;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 300s;
    chunked_transfer_encoding on;
}

[BOUNDARY]Also check Cloudflare Tunnel / proxy 100-second idle timeout if exposing endpoints externally.

The model hit max_tokens (or num_predict) mid-JSON-object. The stream closed cleanly with HTTP 200 and [DONE], leaving half-written JSON.

1. Verify choice.finish_reason === "stop" before running JSON.parse.
2. For reasoning models (DeepSeek-R1), strip <think>...</think> blocks and budget at least 2048 tokens for output.

if (lastChunk.choices?.[0]?.finish_reason === 'length') {
  throw new Error('Output truncated by max_tokens limit before JSON closed');
}

[BOUNDARY]Combining response_format: { type: "json_object" } with low max_tokens still truncates if limit is reached!