A long chat's real cost is the history you resend on every turn

A language model remembers nothing between one call and the next. To make it "remember" the conversation, you send it the whole conversation on every turn. That's where the hidden cost lives.

The math that isn't linear

On turn 1 you send one message. On turn 2, two. On turn 50, fifty. Every new turn pays again for all the previous ones.

That's why the total text sent doesn't grow like the chat: it grows much faster. Doubling the number of turns multiplies the total by almost four.

Measured on simulated conversations, counting the history characters the model receives across all turns:

• 20 turns: 157 thousand characters in total.
• 60 turns: 1.4 million.
• 200 turns: 16.2 million. The last turn alone sends 162 thousand.

A support chat that drags on, an assistant that keeps someone company all afternoon, a widget that never closes: all three live in the expensive part of that curve.

A window plus a summary

The way out isn't erasing memory, it's no longer sending everything raw:

1. A window. Only the newest messages go through, with two ceilings: count (say, 40 messages) and size (say, ~32,000 characters). Whichever comes first.
2. The window starts on a message from the person. If it starts halfway through an assistant reply, the model reads an answer without its question.
3. What was left out is summarized in a single message, labeled as a summary so the model knows it isn't verbatim.

With that, the 200-turn chat goes from 16.2 to 3.4 million characters, and the last turn from 162 thousand to 18 thousand. The 20-turn one doesn't change: it never reaches the ceiling, so short conversations stay exactly the same.

A summary that doesn't cost another call

Another model could write the summary, but that adds a call, latency and a new source of errors. There's a cheaper option: an extractive summary. It's built from text that already exists: the person's first request and the start of each request that was left out, with its own size ceiling.

It has three advantages: it costs nothing, it's always the same for the same chat (so you can test it), and it doesn't repeat what the assistant said, which is exactly what a model tends to copy without checking. If you ever need a written summary, you swap that one piece.

Before turning it on

• It can be switched off without deploying code. An environment variable that, set to zero, brings back the old behavior.
• Measure before and after, with conversations of different lengths. If you don't measure the 20-turn one, you don't know whether you broke the short ones.

The user still sees the whole conversation on screen. The only thing that changes is what travels to the model on each turn.

Try it with your own numbers: add up the characters your assistant sends in a 200-turn chat, turn on the window and add them up again. If you want help measuring it, write to us through the chat on this page.