Running out of Room

"It forgot what I told it" is not a glitch. Here is how tokens and the context window set a hard limit on what AI can hold at once, and how to work around it.

The model isn't forgetting. It's making room.

The Conversation That Loses Its Own Beginning

You're twenty messages into a long exchange with an AI. You gave it clear instructions at the start. Somewhere along the way it drifts, contradicts an earlier point, or asks for something you already provided. It feels like the tool stopped paying attention.

It did, in a specific and predictable way. Every AI has a limit on how much it can hold at once, and when a conversation runs past that limit, the earliest parts fall out of view. The model isn't being careless. It's run out of room.

That room has a name, the context window, and it's measured in a unit we've met before.

Measured in Tokens

We defined tokens back in Issue 15, so here is the short version. A token is the basic chunk of text an AI processes, roughly a word or a piece of one. Everything the model reads and writes is counted in tokens. They are the currency of AI, the unit behind pricing, capability, and the limit we are about to discuss.

Hold onto one number. A rough rule is that 1,000 tokens is about 750 words. Keep it handy, because it makes the next part concrete.

Think of It as a Desk

Here is the mental model to help simplify it.

A context window isn't the AI's long-term memory. It's a desk. Everything the model can see and use right now, your instructions, the conversation so far, any documents you attached, sits on that desk at the same time. The desk has a fixed size, measured in tokens.

While there is space, the model sees everything. Once the desk fills up, something has to come off to make room for what is new, and it is usually the oldest material. Your careful setup from the start of the conversation is the first thing pushed off the edge.

The sizes are large and growing. A model with a 200,000 token window can hold roughly 150,000 words on its desk, a small book. A 1 million token window holds around 750,000 words. That sounds limitless until you feed it a few long documents and a sprawling conversation, and the edges start dropping off.

Critically, the AI doesn't warn you when this happens. It simply proceeds with whatever is still on the desk, which is why it can contradict something you said an hour ago with total confidence.

🚀 MARTY SAYS

"In space, everything you carry has weight, and weight is limited. Bring something new aboard and something old has to go out the airlock."

Who Decides How Big

You don't set the size of the context window. The company that builds the model does, and it's a deliberate trade-off against cost. Bigger windows are more expensive to run, so they tend to sit behind the higher tiers. The free version of a tool often comes with a smaller window, while paid plans, premium models, and enterprise contracts unlock larger ones. More memory, in practice, is something you buy. It's usually why the paid version can handle a long document when the free one can't.

Bigger Is Not Always Better

The obvious wish is a bigger desk, and the industry has been racing to deliver one. But size isn't a free win.

Bigger windows cost more, because you pay by the token and a full window is a lot of tokens. They can also slow responses down. And there is a subtler problem: models often pay less reliable attention to material in the vast middle of a very full window, so simply cramming everything in can quietly hurt quality. A smaller, well-managed window frequently beats a giant, cluttered one. What's on the desk matters more than how big the desk is.

MANAGING THE DESK

Working with the context window is a skill, and a few habits cover most situations. For anything important, put your key instructions at the very start and repeat the critical ones near the end, where the model's attention is strongest. When a long conversation starts drifting, do not fight it. Start a fresh chat and paste in a short summary of what matters, giving the model a clean desk with only the essentials. Attach only the documents relevant to the question, not everything you have. And if a tool advertises a huge context window, treat it as a ceiling, not a promise of perfect recall across every word.

Safe Harbor: Three Things You Can Do This Week

Find your tool's context window. Look up the token limit for the AI you use most. It's usually in the tool's docs or a quick search away ("[your tool] context window"), since most apps don't show it in the interface. Multiply the token number by about 0.75 for a rough word count. Putting a real number on the limit makes it something you can plan around instead of a surprise.
Start fresh when it drifts. When a long chat starts forgetting or contradicting itself, that's the window filling up. Don't fight it. Open a new conversation and paste a short summary of what matters, giving the model a clean desk with only the essentials.
Front-load what matters. Put your most important instruction in your very first message rather than burying it in the middle of a long back-and-forth. Position affects how reliably the model holds onto it.

Next week: how the newest AI models "think" before they answer. Reasoning models take a visibly different approach to hard problems, and understanding what is really happening under the label separates genuine capability from clever marketing.