Keeping an SSE stream alive while the model thinks: headers up front and a comment every 15 s

The stream works perfectly until the model thinks a little longer than usual.

In our API, the streaming chat endpoint (/chat-apiKeyAuth/stream) has a guard: if the request hasn't sent headers after 60 s, it answers 504. One day in September, 4 of 44 turns came back as 504. The case we measured took about 95 s to get the model's first chunk. The turn was still alive; the guard killed it.

The cause was in our controller: it set the SSE headers with setHeader and nothing else, so they didn't go out until the first res.write, and the first write only happened with the first token.

1. Send the headers on open

export function abrirSSE(res, { pingMs = 15_000 } = {}) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.flushHeaders(); // the 200 goes out now, not with the first token
  // ...
}

flushHeaders() sends the 200 and the headers right away. From then on, any timeout that checks "has it responded yet?" treats the request as answered.

2. An SSE comment every 15 s

  const vivo = () => !res.writableEnded && !res.destroyed;
  const timer = setInterval(() => {
    if (vivo()) res.write(': ping\n\n');
    else clearInterval(timer);
  }, pingMs);
  res.on('close', () => clearInterval(timer));
  res.on('finish', () => clearInterval(timer));

In the SSE spec, a line that starts with : is a comment: EventSource and any client that follows the spec ignore it. The client gets no event, but the connection carries traffic, so an intermediate proxy doesn't treat it as dead.

Turning it off matters as much as turning it on. When the client leaves or the turn ends, the interval is cleared; otherwise it would keep writing to a closed socket.

3. The SDK timeout doesn't measure what you think

Seeing a 95-second turn, the natural reaction is to give the model's SDK a longer timeout. Reading the code, both the OpenAI SDK (v4) and the Anthropic SDK build their timeout like this:

const timeout = setTimeout(() => controller.abort(), ms);
// ...
return this.fetch(url, fetchOptions).finally(() => clearTimeout(timeout));

clearTimeout runs when fetch resolves, and fetch resolves when the response headers arrive. With streaming, that's at the very start. The SDK timeout caps the connection, not the whole turn.

What we did change, for streaming only: the model client runs with a 45 s timeout and 1 retry. LangChain retried 6 times by default, and with a stream open on the user's side, six retries is a long time to stare at an empty screen. Non-streaming calls are untouched, because there the headers arrive with the full response.

How we tested it

A test with fake timers spins up an LLM that takes 70 s to produce its first chunk:

• The response is 200 with Content-Type: text/event-stream, never the 504.
• 4 pings go out before the first chunk (70 / 15).
• The stream ends with data: [DONE], and once it's done no ping keeps writing.

Against the branch before the fix, the same test returned 504. A second test covers a client that disconnects: the ping stops.

In CognitionDesk

This is how /chat-apiKeyAuth/stream runs today, the endpoint used by the widget and by API-key integrations. If your stream gets cut while the model is thinking, write to us through the chat on this page.