Hooks & Middleware
Intercepting and enhancing AI requests with hooks and middleware.
Last updated: July 2026
An incoming request enters the agent’s lifecycle.
In Generative AI systems, middleware is a modular interception layer that sits inside an AI agent's execution loop. It lets you add custom logic before, during, and after key agent actions (like calling the language model or running a tool) without rewriting the core agent code. Hooks are the specific named attachment points within that middleware layer. Think of them as plug-in sockets in the agent lifecycle: before_model, after_model, pre_tool, post_tool, before_agent, and after_agent.
“A raw LLM call is unpredictable and unguarded. Without middleware, adding safety guardrails, logging, cost controls, PII redaction, or human approval means tangling that logic directly into your agent code. Middleware and hooks give you a clean, composable way to add production-grade behavior without touching your model or prompt logic.”
| Component | What it does | Example |
|---|---|---|
Hook | Named attachment point in the agent lifecycle | before_model fires before every LLM call |
Middleware class | Bundles multiple hook methods into a reusable unit | SummarizationMiddleware manages token limits automatically |
AgentState | Shared dictionary passed through all hooks | Contains all messages, tool choices, metadata, and jump flags |
next() / passthrough | Returning None to let the agent continue normally | A logging hook that observes then passes through |
Short-circuit / Jump | Returning a state update with jump_to to stop the loop early | A safety hook that returns 'jump_to': 'end' when a policy violation is detected |
modify_model_request | Temporarily mutates the LLM call without altering permanent state | Swapping the model, rewriting the prompt, or changing available tools for one request |
wrap_tool_call | Wraps each tool execution for retries, caching, or confirmation | HumanInTheLoopMiddleware pauses before a destructive tool runs |
Full assembled example
Registered stack: [LoggingMiddleware, SafetyMiddleware, SummarizationMiddleware]. before_model fires in order: Logging → Safety → Summarization. after_model fires in REVERSE order: Summarization → Safety → Logging. A safety violation at before_model returns jump_to: 'end', skipping the LLM and all downstream hooks.
Core Principles
- 1Single Responsibility per Middleware - A PIIMiddleware only redacts sensitive data. It does not also log token counts. Those belong in separate middleware in the same stack.
- 2Execution Order Is Non-Negotiable - If you register [LoggingMiddleware, SafetyMiddleware, SummarizationMiddleware], the after_model order is Summarization → Safety → Logging.
- 3State Is the Shared Contract - Attach the user session ID as state['app_context']['user_id'] rather than overwriting a built-in key like state['messages'].
- 4Return None to Continue, Return State to Act - A hook that returns None does nothing. A hook that returns a partial state dictionary triggers a merge and can reroute the loop.
- 5Fail Safe, Not Silent - Wrap hook logic in try/except and always either raise the exception (to halt the agent) or log and return None (to proceed with observation).
- 6Composability Over Monoliths - Combine SummarizationMiddleware + HumanInTheLoopMiddleware + a custom ComplianceAuditMiddleware in a single create_agent call.
Before vs. after middleware
Before
A hospital where doctors phone pharmacies directly with prescriptions. No pharmacist check, no drug interaction review, no dosage validation. To add a safety step, you'd have to retrain every doctor.
After
Every prescription passes through a pipeline: Doctor writes prescription → Pharmacist checks drug interactions → Safety system validates dosage → Insurance verifies coverage → Patient receives medication. Each station handles one concern.
Summarization Middleware
Keeps context window limits in check by summarizing long conversations before the LLM is called
Prompt
A customer support agent has been running for 40 turns. SummarizationMiddleware fires in before_model, compresses older messages into a summary, and keeps recent messages intact. Code: SummarizationMiddleware(model='openai:gpt-5.5-mini', max_tokens_before_summary=3000, messages_to_keep=10)
Output
Context managed transparently without changing any model or prompt logic.
Human-in-the-Loop Middleware
Pauses agent execution and requires explicit human approval before a specified tool runs
Prompt
A DevOps agent is about to run delete_database. With HumanInTheLoopMiddleware, the agent freezes and sends a confirmation request. Code: HumanInTheLoopMiddleware(interrupt_on={'send_email': True})Output
Agent pauses; human approves or rejects before execution continues.
PII Detection Middleware
Detects and redacts Personally Identifiable Information from prompts before they reach the LLM, and from tool outputs before they return to the model
Prompt
A healthcare agent searches patient records. A post_tool hook intercepts the result and replaces SSN: 123-45-6789 with SSN: ***-**-**** before the LLM sees it.
Output
Model never processes raw PII, so compliance is maintained automatically.
Guardrails / Input Safety Middleware
Intercepts user input before the agent starts, then blocks or transforms harmful, off-topic, or policy-violating requests
Prompt
A financial chatbot should never discuss competitors. A pre_run guardrail checks every message against a topic blocklist and returns a polite deflection, without calling the LLM.
Output
Policy-violating request blocked before any compute is consumed.
Model Fallback / Retry Middleware
Wraps the LLM call to handle failures by retrying with the same model or falling back to an alternative
Prompt
Your agent calls GPT-5.5, which returns a 429 rate-limit error. ModelFallbackMiddleware catches it and retries with claude-sonnet instead.
Output
Smooth failover with no user-visible interruption.
Tool Confirmation Hooks
A pre_tool hook intercepts every tool call before execution and can approve, deny, or request human confirmation
Prompt
A file management agent is given delete_file. A pre-tool hook checks: Is this file in the /protected directory? If yes, it returns decision: 'deny'. Code: create_confirmation_hook(tool_names=['delete_file', 'move_file'])
Output
Destructive actions are blocked or routed for approval automatically.
Best Practices
- 1Keep Each Middleware Focused on One Concern - Use three separate middleware: PIIMiddleware, LoggingMiddleware, and ModelCallLimitMiddleware. That beats one mega-middleware that tries to do all three.
- 2Return None for Observation, State for Intervention - A logging middleware always returns None. A safety middleware returns {'messages': [...], 'jump_to': 'end'} when it detects a policy violation.
- 3Always Register Middleware in the Right Order - Register safety/guardrail middleware before summarization. That way harmful inputs are blocked before tokens are spent on summarization.
- 4Test Middleware in Isolation - Unit-test PIIMiddleware by passing mock AgentState objects containing fake SSNs and checking that the output is masked, without spinning up a real LLM.
- 5Use wrap_tool_call for Retries, Not Business Logic - Put retry logic with exponential backoff in a wrap_tool_call hook for network-bound tools. Do not put retry logic inside the tool function itself.
- 6Scope Hooks to Specific Tools When Possible - You can scope a hook to fire only for 'send_email' instead of all tools. Hook(event_type=EventType.PRE_TOOL, callback=validate_email, tool_names=['send_email'], priority=10)
- 7Handle Errors Gracefully, Never Swallow Silently - Wrap all hook logic in try/except. On failure, either raise (halt the agent with a clear error) or log and return None (degrade gracefully).
Common Mistakes
| Mistake | Real-World GenAI Scenario | Fix |
|---|---|---|
| Silently swallowing hook errors | A before_model safety check throws an exception that gets swallowed, so the agent proceeds with no guardrails active | Always re-raise or log and return None; never use an empty except block |
| Putting business logic in middleware | Auth logic AND summarization AND PII redaction all crammed into one before_model hook | One middleware per concern; compose them in the middleware list |
| Wrong execution order | Logging middleware registered after safety middleware, so blocked requests never appear in logs | Register logging first, then safety, then summarization |
| Confusing callbacks with hooks | Using LangChain callbacks thinking they can modify agent state or reroute the agent | Callbacks = observe only; hooks/middleware = intercept and mutate |
| Not scoping hooks to specific tools | A heavy validation hook runs before every single tool call, including cheap read-only ones | Use tool_names=[...] to scope hooks to only the tools that need them |
| Forgetting that after_model runs in reverse order | Developer assumes all three after_model hooks run in the same order as before_model hooks | Understand the 'onion' model: after_model fires in reverse registration order |
| Blocking async hook pipelines | A before_model hook makes a synchronous HTTP call to a compliance API, blocking the entire async event loop | Use async def hooks and await I/O operations; wrap sync calls with asyncio.to_thread() |
Use Cases
| Use Case | Real-World GenAI Example | Best Hook/Middleware Approach |
|---|---|---|
| Compliance & Regulatory | A banking chatbot must never advise on unlicensed securities | before_agent guardrail hook with topic blocklist |
| PII Protection | A healthcare agent searches patient records before sending them to the LLM | post_tool hook masks SSNs, emails, phone numbers before LLM sees results |
| Cost Control | An internal code assistant is burning $10k/month due to runaway agent loops | ModelCallLimitMiddleware caps LLM calls per session; ToolCallLimitMiddleware limits tool use |
| Human Oversight | A finance agent is about to wire $50,000 via an API tool | HumanInTheLoopMiddleware pauses and requires CFO approval via a confirmation dialog |
| Context Management | A legal research agent accumulates 80 turns of messages, exceeding token limits | SummarizationMiddleware auto-compresses older messages before the next LLM call |
| Observability & Debugging | An agent is underperforming in production but you can't tell why | before_model + after_model logging hooks emit structured traces to LangSmith or OpenTelemetry |
| Model Resilience | GPT-5.5 returns rate-limit errors during peak traffic | ModelFallbackMiddleware retries with Claude or a cached response |
| Input Sanitization | A coding assistant receives prompt-injection attempts via user input | before_agent safety hook detects injection patterns and short-circuits before the LLM sees the input |
Logging + Message Limit Guard (LangChain, Decorator-Style)
Prompt
@before_model
def log_before(state: AgentState, runtime: Runtime) -> dict | None:
print(f'[LOG] Calling LLM with {len(state["messages"])} messages.')
return None
@before_model(can_jump_to=['end'])
def enforce_message_limit(state: AgentState, runtime: Runtime) -> dict | None:
if len(state['messages']) >= 10:
return {'messages': [AIMessage('Conversation limit reached.')], 'jump_to': 'end'}
return None
@after_model
def log_after(state: AgentState, runtime: Runtime) -> dict | None:
print(f'[LOG] LLM replied: {state["messages"][-1].content}')
return None
agent = create_agent(model='openai:gpt-5.5', tools=[weather_tool], middleware=[log_before, enforce_message_limit, log_after])Output
[LOG] Calling LLM with 1 messages. [LOG] LLM replied: I'll check the weather in London for you. [LOG] Calling LLM with 3 messages. [LOG] LLM replied: The weather in London is 14°C and cloudy.
Summarization + Human-in-the-Loop (LangChain, Class-Based)
Prompt
agent = create_agent(
model='openai:gpt-5.5',
tools=[read_email, send_email, delete_email],
middleware=[
SummarizationMiddleware(model='openai:gpt-5.5-mini', max_tokens_before_summary=3000, messages_to_keep=10),
HumanInTheLoopMiddleware(interrupt_on={
'send_email': {'description': 'About to send an email - approve?'},
'delete_email': {'description': 'About to permanently delete an email - approve?'}
})
]
)
response = agent.invoke({'messages': [{'role': 'user', 'content': 'Draft and send a reply to the latest invoice email.'}]})Output
1. Agent reads email inbox → no approval needed (read_email not in interrupt_on) 2. Agent drafts reply → SummarizationMiddleware trims context if needed 3. Agent calls send_email → HumanInTheLoopMiddleware fires: APPROVAL REQUIRED | Tool: send_email | To: billing@vendor.com | Subject: RE: Invoice #4821 | [Approve] [Reject] 4. Human clicks Approve → agent sends email
Pre-Tool PII Masking + Input Validation (ragbits)
Prompt
async def mask_ssn(tool_call: ToolCall, tool_return: ToolReturn) -> ToolReturn:
if tool_call.name != 'search_user':
return tool_return
if isinstance(tool_return.value, dict) and 'ssn' in tool_return.value:
masked = tool_return.value.copy()
masked['ssn'] = '***-**-****'
return ToolReturn(masked)
return tool_return
async def validate_email(tool_call: ToolCall) -> ToolCall:
if tool_call.name != 'send_notification':
return tool_call
email = tool_call.arguments.get('email', '')
if not re.match(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}#x27;, email):
return tool_call.model_copy(update={'decision': 'deny', 'reason': f'Invalid email format: {email}'})
return tool_callOutput
search_user returns: {name: 'Alice Smith', ssn: '123-45-6789', email: 'alice@example.com'}
After post-tool hook: LLM sees {name: 'Alice Smith', ssn: '***-**-****', email: 'alice@example.com'}
send_notification with bad email: decision: deny - Invalid email format: not-an-email. Tool never executes.Tools & Platforms
| Tool / Platform | Best For | Link |
|---|---|---|
| LangChain (v1+) | Full middleware stack: before_model, after_model, wrap_tool_call, built-in prebuilt middleware | https://docs.langchain.com/oss/python/langchain/middleware/overview |
| LangGraph | Stateful agent workflows; middleware hooks run inside compiled LangGraph as nodes/subgraphs | https://langchain-ai.github.io/langgraph/ |
| ragbits | Open-source Python framework with pre_run, pre_tool, post_tool, post_run lifecycle hooks | https://github.com/deepsense-ai/ragbits |
| LangSmith | Observability platform for LLM agents; Prompt Registry for versioned prompts | https://smith.langchain.com/ |
| NVIDIA NeMo Guardrails | Declarative safety/topic guardrails as middleware for LLM pipelines | https://github.com/NVIDIA/NeMo-Guardrails |
| Guardrails AI | Output validation and structured output enforcement; after_model equivalent | https://www.guardrailsai.com/ |
| OpenTelemetry for GenAI | Observability standard for LLM applications; emit traces and metrics | https://opentelemetry.io/docs/specs/semconv/gen-ai/ |
LangChain Middleware Overview
Full middleware stack documentation with examples
LangChain Prebuilt Middleware Reference
Built-in middleware for summarization and HITL
ragbits Agent Hooks Documentation
Open-source hooks system with PRE/POST tool hooks
NVIDIA NeMo Guardrails Docs
Declarative safety guardrails as middleware
Frequently asked questions
What is middleware in AI systems?
Middleware in AI systems is a modular layer that intercepts and enhances AI requests by allowing developers to add custom logic at various points in an AI agent's execution loop. It enables actions like logging, safety checks, or PII redaction without altering the core agent code.
How do hooks function within AI middleware?
Hooks are specific attachment points within AI middleware that allow developers to insert custom logic at key stages in an AI agent's lifecycle. They act like plug-in sockets, enabling actions before and after model calls or tool executions, facilitating easy integration of additional functionalities.
What are the benefits of using middleware and hooks in AI?
Using middleware and hooks in AI provides a structured and clean way to implement production-grade behaviors such as safety checks, logging, and compliance without modifying the core model or prompt logic. This modular approach helps maintain code clarity and reduces the risk of errors.
How does Summarization Middleware help in AI systems?
Summarization Middleware helps manage context window limits by summarizing long conversations before the AI model is called. This ensures that older messages are compressed, keeping the most recent interactions intact, which is crucial for long-running agents or multi-turn chat assistants.
What is the role of Human-in-the-Loop Middleware?
Human-in-the-Loop Middleware pauses AI agent execution and requires explicit human approval before a specified tool runs, especially for actions that are destructive, financial, or irreversible. This middleware ensures that critical operations are reviewed by a human, adding an additional layer of oversight and safety.