04 // DIAGNOSISStreaming / APINVIDIA GPUApple Silicon Mac~8 min read & lab
Post-HTTP-200 SSE Stream Errors: Failure Conditions & Recovery Boundaries
Why simple HTTP status-code tables fail in streaming LLM apps: handling mid-stream SSE error events, proxy idle cuts, and max_tokens JSON truncation.
[KEY VERIFIED TAKEAWAY / TL;DR]In Tinker AI Lab's reproducible verification of "Post-HTTP-200 SSE Stream Errors: Failure Conditions & Recovery Boundaries" (tested on Node.js 22.13+ SSE Client · Applicable to Claude API, OpenAI API, and Local Ollama / vLLM with Node.js 22.13+, Anthropic Messages SSE, OpenAI Chat Completions SSE, verified 2026-10-23): Why simple HTTP status-code tables fail in streaming LLM apps: handling mid-stream SSE error events, proxy idle cuts, and max_tokens JSON truncation. Offline capability is classified as "Applies to Both Local & Cloud Streaming APIs" (Includes local zero-cost SSE guard client to test your parser against mid-stream error events.), with key operational boundary: As documented by Claude & OpenAI, once HTTP 200 OK headers are flushed to open an SSE stream, an error occurring mid-generation CANNOT change the HTTP status code.
REPRODUCIBILITY MANIFEST // TESTED SPECS & LIMITS
Last Verified: 2026-10-23Tested Hardware & OS
Node.js 22.13+ SSE Client · Applicable to Claude API, OpenAI API, and Local Ollama / vLLM
Pinned Stack & Model Versions
Node.js 22.13+Anthropic Messages SSEOpenAI Chat Completions SSE
Offline & Network Boundary
Applies to Both Local & Cloud Streaming APIs
Includes local zero-cost SSE guard client to test your parser against mid-stream error events.
Known Limits & Failure Modes
- As documented by Claude & OpenAI, once HTTP 200 OK headers are flushed to open an SSE stream, an error occurring mid-generation CANNOT change the HTTP status code.
- Retrying a partially streamed response that already triggered external side effects without an idempotency key causes duplicate writes.
Hit an edge-case failure?Open Troubleshooter →
1. Why Checking `if (!res.ok)` Is Not Enough for Streaming LLMs
In standard REST APIs, `HTTP 200` means the request succeeded. In Server-Sent Events (SSE), `HTTP 200` only means the connection handshake succeeded before token #1 was generated. If the server hits a CUDA OOM on token #180, or an upstream provider experiences an overloaded worker mid-generation, it emits an event frame like `event: error` or `data: {"error": {"type": "overloaded_error", ...}}` and closes the socket. Naive clients silently treat the truncated text as a complete answer!
Table 1: Post-HTTP-200 SSE Failure Modes, Wire Frames & Recovery Boundaries
| Mid-Stream Failure Mode | Wire Frame After HTTP 200 | Naive Client Behavior | Correct Recovery Action |
|---|---|---|---|
| Upstream Overload / CUDA OOM | data: {"type":"error","error":{...}} | Ignores error field; saves half sentence | Throw on event.error; discard partial buffer & retry |
| Reverse Proxy 60s Idle Cut | TCP FIN/RST without [DONE] marker | Loop exits cleanly on done===true | Require explicit [DONE] flag before marking completed |
| max_tokens / num_predict Hit | finish_reason: "length" + [DONE] | Crashes in JSON.parse(output) | Check finish_reason === "stop" before parsing JSON |
| Stalled Socket (Half-Open TCP) | Zero bytes indefinitely | await reader.read() hangs forever | Reset per-chunk watchdog timer & call reader.cancel() |
Three Mandatory Boundaries for Production SSE Readers
// 1. Inspect every SSE JSON frame for mid-stream errors after HTTP 200:
if (event.error || event.type === 'error') {
throw new Error(`Mid-stream SSE error: ${JSON.stringify(event.error || event)}`);
}
// 2. Check finish_reason before JSON.parse:
if (event.choices?.[0]?.finish_reason === 'length') {
throw new Error('Truncated by max_tokens before JSON closed');
}
// 3. Enforce per-chunk idle timeout with reader.cancel()2. Distinguishing First-Token Latency (TTFT) from Inter-Chunk Stall
When sending a 24K-token prompt to a local model or a reasoning model (like DeepSeek-R1), the server may take 30–75 seconds of silent prefill computation before emitting the first SSE chunk. Set a generous **Initial Prefill Timeout (60–120s)** before token #1, and tighten the **Inter-Token Chunk Timeout (15s)** once tokens begin flowing.
3. Transactional Buffering & Idempotent Retry Boundaries
Never mutate database rows or trigger external webhooks incrementally inside the SSE chunk loop. Accumulate chunks in an in-memory buffer (`fullText`), verify both `sawDoneMarker === true` and `finishReason === "stop"`, and only then commit to your database using a deterministic idempotency key (`hash(requestId + promptHash)`).
FIELD Q&A // 常见实测疑问与边界解答
Frequently Asked Questions & Edge Cases
Q: Does the official OpenAI / Anthropic SDK handle mid-stream SSE error events automatically?
Recent versions of official SDKs throw an `APIError` when an SSE `error` event frame arrives, but they do NOT automatically discard partial text that your `for await (const chunk of stream)` loop already wrote to your UI or database. Always buffer transactional writes until the stream completes with `finish_reason === "stop"`.
Q: Why does Nginx cut off my SSE stream at exactly 60 seconds even when tokens are still generating?
Nginx defaults to `proxy_buffering on` and `proxy_read_timeout 60s`. When buffering is enabled, chunks may sit in Nginx buffers, or a long prefill exceeds 60s between packets. Set `proxy_buffering off; proxy_cache off; proxy_read_timeout 600s;` on the location block.
REPRODUCIBLE ARTIFACT // 配套可参数化附件
sse-stream-guard.mjs ↗Resilient LLM SSE Stream Reader (Handles Post-HTTP-200 Mid-Stream Errors)
Detects mid-stream error frames after HTTP 200 OK, idle socket stalls, and premature finish_reason truncation.
Edit variables above to update the configuration live
// Tinker AI — Resilient SSE Stream Reader with Mid-Stream Error Detection
const API_URL = "http://127.0.0.1:11434/v1/chat/completions";
const IDLE_TIMEOUT_MS = Number("15000");
export async function consumeLlmStream(payload) {
const res = await fetch(API_URL, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ...payload, stream: true }),
});
if (!res.ok || !res.body) {
throw new Error(`Pre-stream HTTP failure: ${res.status}`);
}
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
let output = "";
while (true) {
const timer = setTimeout(() => reader.cancel("SSE chunk idle timeout"), IDLE_TIMEOUT_MS);
const { value, done } = await reader.read().finally(() => clearTimeout(timer));
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop() ?? "";
for (const raw of lines) {
const line = raw.trim();
if (!line.startsWith("data:")) continue;
const data = line.slice(5).trim();
if (data === "[DONE]") return { output, completed: true };
const event = JSON.parse(data);
// Catch mid-stream error frame sent AFTER HTTP 200 OK
if (event.error || event.type === "error") {
throw new Error(`Mid-stream SSE error after HTTP 200: ${ JSON.stringify(event.error || event) }`);
}
const delta = event.choices?.[0]?.delta?.content ?? "";
output += delta;
}
}
return { output, completed: false };
}
More Reproducible Engineering Guides
View All 42 Guides →API & Deployment Troubleshooting
Verified 2026-10-23 →Container Cannot Connect to Host Model: Minimal Reproduction & Diagnostic Steps
API & Deployment Troubleshooting
Verified 2026-10-23 →Fixing Local LLM HTTP 403 CORS, Reverse Proxy Host Headers & LAN Bearer Authentication
API & Deployment Troubleshooting
Verified 2026-10-26 →