Context Engineering
Structuring context effectively for AI models.
Last updated: July 2026
Listen: Context Engineering (student & teacher)StudentTeacher
0:000:00
Context engineering is the practice of deciding what an AI system should see at each step: the information, instructions, tools, memory, and references that help it produce the best result. It is broader than just the wording of a prompt. It covers the whole context state, including system instructions, tool definitions, message history, external data, and anything pulled in at runtime.
“Large language models do not get better just because you give them more tokens. The goal is the smallest high-signal context that still supports the task.”
| Component | What it does | Example |
|---|---|---|
System Instructions | Define the role, goals, constraints, and the shape of the output | A coding agent gets 'fix the bug, run targeted checks, and return changed files plus validation status' |
Tools | Expand what the model can do, and work best when each one has a clear purpose and little overlap | A repo agent has separate search, file-read, and test-run tools instead of one giant workspace tool |
Retrieval Layer | Keeps lightweight references and pulls in relevant material right when it is needed | A research assistant stores article URLs, then fetches only the two most relevant sources for the current claim |
Short-term Memory | Tracks the current thread or state across steps within a session | A troubleshooting assistant remembers the device model, failed steps, and current error code during the session |
Long-term Memory | Stores lasting facts or progress notes outside the active window | A sales copilot stores a customer's renewal date and preferred contract terms for future conversations |
Compaction | Summarizes the important decisions and drops stale content to keep the context smaller | A coding agent keeps architecture decisions and open bugs, but drops old raw tool logs |
Sub-agents / Orchestration | Hands focused tasks to specialized sub-agents and takes back only their summaries | A market research agent asks one sub-agent to analyze competitors and another to scan regulations, then combines the results |
Full assembled example
Role: Customer support assistant. Goal: Resolve the billing issue. Tools: issue_refund(account_id, amount). Context: Account tier: Pro, Renewal: 2026-05-20. Retrieved policy: Section 4.2 - refundable within 7 days of renewal. Output: JSON with decision, reason, action_taken, missing_info.
Core Principles
- 1Use the minimum useful context - A contract assistant does better with the relevant clause and the party names than with the entire deal room pasted in
- 2Structure beats dumping - Sectioned prompts, clear tool contracts, and explicit output requirements cut down on confusion far more than one big paragraph
- 3Retrieve just in time - A repo agent reads only the failing test and the touched files first, then expands if it needs to, rather than loading everything upfront
- 4Separate working memory from durable memory - 'Customer asked for a refund today' is thread state. 'Customer is on the enterprise plan' belongs in durable memory
- 5Use examples selectively - Three strong, clear examples of 'good triage output' usually work better than twenty loosely related ones
- 6Define success and stop rules - A research agent stops once it has two citable sources for the main claim, instead of searching forever
- 7Evaluate and refine - A team compares ticket-resolution accuracy before and after changing the retrieval rules, rather than going on gut feel
Before vs. after context engineering
Before
A support bot gets one long prompt plus a pasted policy document, and answers inconsistently.
After
The bot gets a short role definition, the relevant policy excerpt, the customer's account facts, a refund tool, and a stopping rule. It decides faster and gives fewer unsupported answers.
Types and Approaches
| Type | Definition + example | Best use case |
|---|---|---|
| Static prompt stack | A mostly fixed instruction set with little runtime retrieval. Example: 'Classify each ticket into billing, bug, or refund using this schema and return JSON.' | Narrow, repeatable tasks where the rules rarely change and outside facts are limited. |
| RAG | The system retrieves external documents and adds them to the context before generating a response. Example: a policy bot fetches the top policy passages before answering a benefits question. | Knowledge-grounded Q&A, internal documentation, enterprise search. |
| Just-in-time retrieval | The agent keeps references (file paths, queries, links) and loads information as it needs it. Example: a code assistant opens only the failing module and the nearby tests first. | Large codebases, research workflows, and tasks where relevance shifts step by step. |
| Memory-based context | Stores thread state or lasting facts outside the active window and recalls them later. Example: an account assistant remembers the subscription tier and prior escalations. | Personalization, long-running assistants, multi-session workflows. |
| Compaction | Summarizes long histories into a smaller, high-fidelity state. Example: a debugging agent compresses old exploration into 'root cause hypotheses, tested fixes, remaining blockers.' | Long-horizon tasks where the full history would get noisy or too large. |
| Sub-agent architecture | A lead agent hands focused work to specialized sub-agents and takes back their distilled outputs. Example: a research lead sends legal review and market scanning to separate workers. | Broad research, multi-step analysis, parallel exploration. |
| Hybrid approach | A blend of up-front retrieval, runtime exploration, and memory. Example: a legal assistant loads the core matter facts first, then fetches relevant precedent as the task narrows. | Real production systems where both speed and adaptability matter. |
Best Practices
- 1Start outcome-first, not instruction-first - Define the goal, the success criteria, the constraints, and the output shape before you add detailed process steps
- 2Keep prompts sectioned - A compliance assistant uses separate sections for jurisdiction, evidence sources, allowed tools, and the final answer schema
- 3Make tools narrow and obvious - Use separate search_docs, get_customer_record, and issue_refund tools instead of one vague assistant_backend tool
- 4Retrieve less, but retrieve better - A support assistant runs one broad search, answers from the top two relevant passages, and skips five more searches that would barely help
- 5Store durable notes outside the window - A delivery operations copilot stores warehouse exceptions in memory and pulls back only today's relevant ones for dispatch decisions
- 6Compact safely - After a long incident-response session, the agent keeps the timeline and the unresolved steps but drops repetitive command output
- 7Use a few canonical examples, not a dump of edge cases - Give three well-formed escalation examples that cover success, failure, and ambiguity, rather than a giant pile of rare cases
- 8Validate with evals - Compare the grounded-answer rate and tool-call count before and after you change your retrieval strategy. Treat context design as something you test
Common Mistakes
| Mistake | Real-world scenario | Fix |
|---|---|---|
| Dumping everything into the prompt | A team pastes an entire handbook, every past chat, and all customer metadata into each call, and the answers get fuzzy. | Trim to the relevant facts, retrieve on demand, and treat tokens as a limited attention budget. |
| Over-specifying brittle steps | A prompt says 'always do A, then B, then C' even when the request clearly calls for a different path. | Define the desired outcome and the stopping rules, then let the model pick an efficient path. |
| Using vague tools | An agent has several overlapping tools that all 'search data,' so tool choice becomes inconsistent. | Give each tool a distinct purpose, clear inputs, and token-efficient outputs. |
| Treating memory as one undifferentiated blob | The assistant mixes short-lived conversation state with lasting customer facts and drags both into every turn. | Keep thread memory separate from long-term memory and retrieve only what the current step needs. |
| Stuffing too many examples | A prompt contains dozens of edge cases, many of them only loosely related to the task. | Use a small set of varied canonical examples that show the behavior you expect. |
| Never compacting long sessions | A multi-hour agent run carries old logs, failed attempts, and obsolete context into later steps. | Summarize the lasting signal, keep the recent working items, and drop stale raw output. |
| Searching too much | A research assistant keeps making retrieval calls after it already has enough evidence. | Set a retrieval budget and stop once the core claim is supported. |
| No evaluation loop | Changes to prompts and retrieval ship on gut feel alone. | Track accuracy, grounding, latency, and tool efficiency with repeatable evals. |
Use Cases
| Use case | Real-world example | Best approach |
|---|---|---|
| Customer support | A support assistant checks the account status, fetches the relevant refund policy, and decides whether to issue a credit. | Outcome-first instructions plus targeted retrieval and action tools. |
| Coding assistant | A repo agent inspects the failing tests, reads only the nearby files, writes a patch, and validates it. | Just-in-time retrieval, compact summaries, and narrow repo tools. |
| Research analyst | An agent explores several sources in parallel, then pulls them together into a short answer. | Sub-agents with summarized outputs and a coordinator. |
| Legal or finance review | A system starts with the core matter facts, then pulls specific clauses as the question narrows. | Hybrid retrieval with careful grounding and selective exploration. |
| Enterprise knowledge base | Employees ask policy or product questions over internal documentation. | RAG with strict source grounding and retrieval budgets. |
| Personal assistant | The assistant remembers lasting user preferences across sessions without bloating each turn. | Keep long-term memory separate from thread memory and recall only the relevant facts. |
| Operations copilot | A logistics assistant tracks task state across many steps and hands off open items between shifts. | Short-term state plus durable notes and periodic compaction. |
1
Support Assistant - Structured Context Engineering
Weak prompt
Answer the customer's refund question. They forgot to cancel and were charged.
Better prompt
Strong prompt
Role: Customer support assistant for a SaaS billing team. Goal: Resolve the user's billing issue using policy and account facts. Success criteria: Decide eligibility, use policy evidence, take allowed action, return JSON with decision, reason, action_taken, missing_info. Constraints: Do not invent policy. Ask for only the smallest missing field. Cite the policy section ID. Context: Account tier: Pro. Renewal date: 2026-05-20. Last payment: paid. Retrieved policy: Section 4.2 - Self-serve annual plans are refundable within 7 days of renewal if account usage is below 5 active seats. Tool: issue_refund(account_id, amount) Output: Return valid JSON only.
Output
{
"decision": "eligible",
"reason": "Eligible under policy section 4.2 - request is within 7 days of renewal.",
"action_taken": "Refund initiated with issue_refund(account_id, amount).",
"missing_info": null
}2
Coding Assistant - Just-in-Time Retrieval
Weak prompt
Fix the failing unit test. [entire repo pasted in]
Better prompt
Strong prompt
Role: Repository debugging assistant. Goal: Fix the failing unit test with the fewest useful file reads. Success criteria: Identify root cause, change only necessary files, run targeted validation, return root_cause, files_changed, validation. Constraints: Read files just in time. Prefer the failing test, touched module, and nearest helper first. If context grows too large, summarize decisions and continue from the summary. Current context: Failing test: tests/test_pricing.py::test_discount_rounding. Recent commit: 'refactor discount formatting'
Output
root_cause: The refactor changed rounding from decimal-half-up to Python default bankers rounding in pricing/discounts.py. files_changed: pricing/discounts.py validation: Re-ran tests/test_pricing.py::test_discount_rounding → passed. Re-ran pricing test subset → passed.
3
HR Policy Assistant - Retrieval Budget + Stop Rules
Weak prompt
Can I carry over unused parental leave into next year? [entire HR handbook pasted in]
Better prompt
Strong prompt
Role: Internal HR policy assistant. Goal: Answer the employee's question using retrieved policy text only. Success criteria: Give a direct answer first. Quote only the relevant rule. If policy is silent, say so clearly. Return answer plus cited source IDs. Constraints: No unsupported advice. No guessing from general HR norms. Stop after enough evidence is found. Retrieved: HR-PL-12: 'Unused parental leave does not roll over across calendar years unless local law requires otherwise.' HR-LOC-7: 'In California, statutory leave rights follow state rules and may override default company carryover policy.'
Output
Direct answer: Usually no. Under HR-PL-12, unused parental leave does not carry over. Important exception: If covered by a local rule (e.g., California - HR-LOC-7), that local rule may apply. Cited source IDs: HR-PL-12, HR-LOC-7
Tools & Platforms
| Tool | Best for | Link |
|---|---|---|
| Anthropic | Practical guidance on agent context, long-horizon work, compaction, tool design, and memory patterns. | https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents |
| OpenAI API | Outcome-first prompting, stopping conditions, retrieval budgets, and structured prompt design. | https://developers.openai.com/api/docs/guides/prompt-guidance |
| LangChain / LangGraph | Short-term and long-term memory patterns for agents, including thread state and persistence. | https://docs.langchain.com/oss/python/concepts/memory |
| LlamaIndex | Building context-aware applications and workflow-driven retrieval or context refinement systems. | https://www.llamaindex.ai/llamaindex |
| Pinecone | Retrieval-augmented generation and external knowledge grounding for production systems. | https://www.pinecone.io/learn/retrieval-augmented-generation/ |
Official Guides
Frequently asked questions
What is context engineering in AI?
Context engineering in AI involves structuring the information, instructions, tools, and memory that an AI system uses to produce optimal results. It goes beyond prompt wording to include system instructions, tool definitions, and runtime data.
How does context engineering improve AI performance?
Context engineering improves AI performance by ensuring that only the most relevant and high-signal information is used, which helps the AI make more accurate decisions. By structuring context effectively, it reduces confusion and enhances the AI's ability to focus on the task.
What are the key components of context engineering?
The key components of context engineering include system instructions, tools, retrieval layers, short-term and long-term memory, compaction, and sub-agents. Each component plays a role in defining the AI's role, expanding its capabilities, and managing information efficiently.
Why is using minimal context important in AI?
Using minimal context in AI is important because it helps avoid overwhelming the system with unnecessary information, ensuring that the AI focuses only on what is crucial for the task. This approach leads to more efficient processing and better task performance.
How can AI systems use memory effectively in context engineering?
AI systems use memory effectively by separating short-term memory, which tracks session-specific information, from long-term memory, which stores lasting facts. This separation allows the AI to recall only relevant information when needed, improving decision-making and personalization.