Context engineering: what your AI agent sees at each step decides what it does
An AI agent doesn't remember your business. It only sees what's in its context at that moment. How to decide what goes in, what comes out, and how to keep focus on long tasks.
A language model doesn’t know your company. At each step, an AI agent decides with what’s in front of it: its instructions, the tools available, the conversation so far and whatever the tools returned. That’s called context, and deciding what goes into it is one of the most important jobs in building an agent.
Anthropic calls it context engineering: choosing and maintaining the best set of information for each step, not just writing a good prompt (Effective context engineering for AI agents, 2025). This is the third note in the series on how to build AI agents.
More context isn’t better context
Intuition says that if you give the agent all the information, it’ll decide better. The opposite happens. As the context window fills up, the model recalls what’s in it less well. Anthropic calls it context rot. The model has an attention budget, and every new token spends a little of it.
The goal, in their words, is to find the smallest set of relevant information that maximizes the chance of the outcome you want. Everything below is a way to apply that idea.
Instructions at the right altitude
There are two ways to write an agent’s instructions badly:
- Too rigid. A chain of “if X happens, do Y” that tries to cover every case. It breaks on the first case nobody foresaw, and it’s costly to maintain.
- Too vague. “Be helpful and give customers great service.” The model doesn’t know what matters to you.
The middle ground is specific about what matters and leaves judgment for the rest. For a support agent, it looks like this:
## Your jobYou help customers of a dental clinic over chat. You book, move and cancel appointments,and answer questions about hours, services and prices.
## What you don't doYou don't give diagnoses or medical advice. You don't promise discounts.
## Where the data comes fromPrices, hours and services always come from the tools, never from memory.If a tool doesn't have the information, say so and offer a person.
## When to hand off to a personComplaints, payments or refunds, and when you don't understand the customer after two tries.Start with short instructions, test them on real cases and add a line only when a case fails. That way every line has a reason.
Examples instead of rules
When you want the agent to answer a certain way, a few well-chosen examples teach more than a long list of rules. Pick examples that differ from one another and show the typical behavior, and don’t try to cover every odd case with one. That’s what tests are for.
Fetch data right when it’s needed
There are two ways to give the agent information. Load it all up front, or give it tools to look it up when it needs it. Anthropic calls the second just-in-time retrieval: the agent keeps lightweight references (an order ID, a document name, a query) and loads the content only at the step that uses it.
It’s slower, but it keeps the context clean. In practice, combining the two works well: load up front what’s always used, like company policies, and let the agent look up the rest, like a specific customer’s history. The lookup tools need to be well made, which is what tools for AI agents covers.
Long tasks: three techniques
When an agent works through many steps in a row, the context fills up sooner or later. There are three techniques for handling it:
| Technique | What it does | When it helps |
|---|---|---|
| Compaction | Summarize the conversation as it nears the limit and continue from the summary. The simplest step is clearing old tool results. | Long, single-thread conversations |
| Note-taking | The agent writes notes to a file or database outside the context and reads them back when it needs them. | Work that spans hours or sessions |
| Sub-agents | A lead agent hands a part to another that works with a clean context and returns a short summary. | Broad research with independent parts |
Notes are the technique I use most. They look like what a person would write before picking a job back up the next day:
## GoalReconcile ACME's October invoices against the bank payments.
## Done- 41 of 47 invoices reconciled.
## Pending- 6 invoices with no matching payment. IDs in pending.csv.
## Decisions- Differences under $1 are accepted as rounding (confirmed by accounting).Sub-agents have a cost worth remembering. Each one spends its own context, and a multi-agent system can use about 15 times the tokens of a chat (Anthropic). Use them when the parts of the work really are independent.
How to know it works
Context engineering isn’t done by eye. Every change to the instructions, the examples or what the tools return gets tested against a set of real tasks. If the result improves, it stays. If not, it goes. The method is in how to evaluate an AI agent.