Photo by Brendan C on Flickr, CC BY 2.0
When an agent answers badly, the first instinct is to give it more. A longer prompt, more retrieved documents, a bigger context window. I did that for months. The bill went up and the answers did not get better.
The problem is not the size of the window. It is what goes into it. Context engineering is the work of deciding what the model sees on each call, instead of deciding how much fits.
I wrote earlier about rethinking LangGraph for multi-agent systems. Picking the right architecture matters, but it does not save a system that feeds the model badly. Three problems come from the context, and all three are fixable before the call is made.
The same job, with everything in the prompt and with only what the task needs
Cost
Tokens add up fastest when agents talk to each other. Every call carries the history again, so a conversation that grew over ten turns is paid for ten times. Adding a second agent doubles it.
Two techniques bring it down. Prompt compression means summarising earlier turns before feeding them back. The agent does not need the transcript. It needs the decisions and the facts they were based on. Output constraints mean saying what the answer should look like. A model told to return three fields returns three fields, and a model left open writes a paragraph around them.
On one project I measured about 40% less spend after this change. The number of calls stayed the same. Each call carried less.
Hallucination
A model fills gaps. Give it context that is loosely related to the question and it will still produce something that has the shape of an answer. The wrong facts come from the retrieval step more often than from the model.
The fix is retrieval that is checked before it reaches the prompt. Similarity search returns what is close, not what is right, so the results need re-ranking and filtering first. Anything that does not answer the question is noise, and noise is what the model reaches for when the real answer is missing.
Role and task framing does the rest. Tell the model what it is, what it is doing, and what it should ignore. A model that knows what to ignore stops treating every chunk as evidence.
The context window
When the input passes the limit, something is dropped. The agent does not report this. It answers as if the missing part never existed, which is the hardest failure to notice, because the answer still reads well.
Context windowing is the technique here. Keep recent context in full, because that is what the current task depends on. Summarise older context. Leave irrelevant context out completely. The window then behaves like a workspace with only the current job on it, rather than a box that everything is poured into until it overflows.
The five techniques
- Prompt compression. Summarise earlier turns into decisions and facts before sending them again.
- Smart retrieval. Re-rank and filter results, and drop anything that does not answer the question.
- Context windowing. Recent context in full, older context summarised, irrelevant context out.
- Role and task framing. State what the model is, what it is doing, and what it should ignore.
- Output constraints. Define the shape of the answer, so the model returns fields instead of prose.
Conclusion
A bigger context window moves the limit. It does not improve the answer, and it makes every call more expensive while it does so. The work that improves the answer is choosing what goes in and checking what comes back.
That work belongs somewhere. In an agent system it lives in the harness, which is the layer that assembles the context and decides what the model may do with it. I wrote about how much of a harness a narrow agent really needs in the harness post.