What a context window is, and why long chats drift
A language model does not remember your last message. The weights do not change between one turn and the next; what makes a conversation feel continuous is that the entire transcript is sent again, from the top, every time you press enter. The context window is the ceiling on how much of that transcript fits.
The model re-reads everything, every turn
A chat interface hides a fairly blunt mechanism. Each time you send a message, the request that goes to the model contains the system prompt, any tool definitions, every earlier message from both sides, anything you pasted along the way, and then your new sentence at the end. The model reads that whole block, produces a continuation, and retains nothing. The next turn rebuilds the block from scratch. What you experience as memory is repetition.
All of it is counted in tokens — fragments of words rather than words — and the window is a token budget that the input and the reply share. A long answer eats into the same allowance the transcript is using. Some of the budget is spent before you type anything: the system prompt sits there, and if the assistant has tools, their descriptions do too.
This is also why a long conversation costs more and feels slower, though those are two different facts. The cost is arithmetic: you are billed for the whole transcript again on every turn, so a chat twice as long costs about twice as much per message. The latency is worse than linear, because attention relates every token to every other token and that part of the work grows with roughly the square of the length — the feed-forward layers, which dominate at ordinary lengths, grow linearly. Production systems soften this; prompt caching lets an unchanged prefix be reused instead of recomputed. But caching changes the bill and the wait, not the ceiling. The tokens still occupy the window, and editing something early invalidates the cached prefix from that point on.
What actually happens when the window fills
Something has to go. Products usually handle this in one of two ways, and the difference matters: the simpler approach drops the oldest turns, the more elaborate one compacts them into a summary and sends the summary instead. There are variants — pinning the system prompt and the opening turns while dropping the middle, or retrieving over the chat's own history — but the result is the same shape. The request that reaches the model is shorter than the conversation you can scroll through.
It is worth being precise about the failure, because the usual word for it is wrong. The model has not forgotten your earlier instruction. It never received it. A person who forgets still knows there was something there and can be reminded; a model handed a truncated transcript answers confidently from what it has, with no signal that anything was removed and no way to ask. From the inside there is nothing that marks the gap as a gap.
Compaction is lossy in a particularly annoying way. Summaries preserve gist and shed detail, and your constraints are detail: the exact file path, the version number, the one library you told it never to use, the client's name spelled the unusual way. The general shape of the task survives; the specifics that made your instructions yours are the first thing to go. That is where most of the drift comes from — not a model changing its mind, but a model working from a thinner brief than you think you gave it.
A large window is not the same as a well-used one
Fitting is not the same as using. The best-known result here is the "Lost in the Middle" work by Liu and colleagues (2023), which found a U-shaped curve: on multi-document question answering and key-value retrieval, accuracy was highest when the needed passage sat near the beginning or the end of the input, and lowest when it sat in the middle. The same shape has been reported repeatedly since, across different models and tasks.
Be careful about what that licenses you to claim. Well established: position within a long input affects whether the model uses the information, and accuracy on many tasks declines as inputs get longer even when the answer is definitely present. Not settled: the size of the effect for any particular current model. It varies a great deal, training specifically for long contexts reduces it, and the underlying mechanism is still argued about. Anyone quoting a single percentage for how much worse the middle is, is quoting one measurement of one model on one task.
One more caveat about the evidence. Many long-context claims rest on needle-in-a-haystack tests, where a distinctive sentence is planted in a wall of filler and the model is asked to find it — close to the easiest long-context task there is. Passing it says little about whether the model can hold a long argument together, weigh a constraint stated on page 4 against a decision made on page 90, or notice that the two contradict. Treat an advertised window size as capacity, not as a promise of attention.
Habits that keep a long chat on course
Start a new conversation for a new task. Old turns are not free background; they compete for attention with what you are asking now. Worse, if the model got something wrong forty turns ago, that mistake is re-read verbatim on every subsequent turn, and the model has no way to tell an earlier mistake of its own from an earlier fact. When a chat has gone circular, opening a fresh one and pasting in only what matters is usually faster than arguing.
Restate the constraints that count in the message that needs them, instead of assuming they survived. It costs one line. Paste the material rather than describing it — "the config we discussed" is not the config, and a model can only work with what is actually in the window. And when you are giving a long document, put the question after it rather than before. The middle is the weak position, and a question at the end is the last thing the model reads before it starts writing.
Arabic reaches the ceiling sooner. Tokenizers trained mostly on English split Arabic into more and smaller pieces, so the same passage costs more tokens; the gap has narrowed with newer tokenizers, but it has not closed. Counted per word the difference looks worse than it is, since an Arabic word packs in grammar that English spreads over several words. The split also lands in the wrong places: where an English tokenizer usually keeps a stem intact, Arabic words are often broken below the morpheme, because the vocabulary was fitted to a corpus in which Arabic was a small minority. Practically, the same document eats more of your budget in Arabic, and a bilingual chat drifts a few turns earlier than its English equivalent — one more consequence of the tokenisation problems that make Arabic awkward for these models generally.
When you need a library instead
The window is scratch space for one conversation. It is temporary, it is not shared with your other chats, and nothing that passes through it changes the model — pasting a manual does not teach the model your manual, and tomorrow's conversation starts blank. Where a product does appear to remember across chats, it is storing text somewhere else and pasting it back into the window; the window itself is still empty at the start of every conversation. "Just use a bigger window" is therefore not an answer to having a lot of material. It only raises the amount you can re-paste.
When the material is larger than the window, or has to outlive the chat, or needs to be shared with colleagues, or has to be answered with a citation, the technique is retrieval: index the documents once, fetch the few passages relevant to each question, and put only those in the window with their sources attached. That is a different architecture with its own difficulties — chunking, ranking, honest citation — and it is covered in its own article here.
A useful rule: if you would have to paste it again tomorrow, it belongs in a library rather than a conversation. MentronX's Firas Brain is the library-shaped half of that split — it answers from an indexed set of documents rather than from whatever is still in the window.
The short version
Treat the context window as a workbench, not a filing cabinet — and notice that nothing in the interface tells you when it is full. The habits that fix drift are unglamorous: shorter conversations, constraints restated where they are needed, sources pasted rather than described. Anything that must survive past the chat needs a library behind it, not a larger window.
More like this