model
, letting it choose tools
to execute, and then finishing when it calls no more tools.
Middleware.beforeModel
: runs before model execution. Can update state or jump to a different node (model
, tools
, __end__
)Middleware.modifyModelRequest
: runs before model execution, to prepare the model request object. Can only modify the current model request object (no permanent state updates) and cannot jump to a different node.Middleware.afterModel
: runs after model execution, before tools are executed. Can update state or jump to a different node (model
, tools
, __end__
)beforeModel
, modifyModelRequest
, afterModel
.
model
must be either a string or a BaseChatModel. Will error if a function is passed. If you want to dynamically control the model, use AgentMiddleware.modifyModelRequest
prompt
must be either a string or None. Will error if a function is passed. If you want to dynamically control the prompt, use AgentMiddleware.modifyModelRequest
preModelHook
must not be provided. Use AgentMiddleware.beforeModel
instead.postModelHook
must not be provided. Use AgentMiddleware.afterModel
instead.summarizationMiddleware
automatically manages conversation history by summarizing older messages when token limits are approached. This middleware monitors the total token count of messages and creates concise summaries to preserve context while staying within model limits.
Key keatures:
model
: Language model to use for generating summaries (required)maxTokensBeforeSummary
: Token threshold that triggers summarizationmessagesToKeep
: Number of recent messages to preserve (default: 20)tokenCounter
: Custom function for counting tokens (defaults to character-based approximation)summaryPrompt
: Custom prompt template for summary generationsummaryPrefix
: Prefix added to system messages containing summaries (default: ”## Previous conversation summary:”)humanInTheLoopMiddleware
enables human oversight and intervention for tool calls made by AI agents. This middleware intercepts tool executions and allows human operators to approve, modify, reject, or manually respond to tool calls before they execute.
Key features:
accept
: Execute the tool with original argumentsedit
: Modify arguments before execution - { type: "edit", args: { action: "tool_name", args: { modified: "args" } } }
ignore
: Skip tool execution and terminate agentresponse
: Provide manual response instead of executing tool - { type: "response", args: "Manual response text" }
toolConfigs
: Map of tool names to their approval settings
requireApproval
: Whether the tool needs human approvaldescription
: Custom message shown during approval requestmessagePrefix
: Default prefix for approval messagesAnthropicPromptCachingMiddleware
is a middleware that enables you to enable Anthropic’s native prompt caching.
Prompt caching enables optimal API usage by allowing resuming from specific prefixes in your prompts.
This is particularly useful for tasks with repetitive prompts or prompts with redundant information.
AgentMiddleware
, which implement one or more of its hooks.
AgentMiddleware
currently provides three different ways to modify the core agent loop:
before_model
: runs before the model is run. Can update state or exit early with a jump.modify_model_request
: runs before the model is run. Cannot update state or exit early with a jump.after_model
: runs after the model is run. Can update state or exit early with a jump.jump_to
key to the state update with one of the following values:
"model"
: Jump to the model node"tools"
: Jump to the tools node"__end__"
: Jump to the end nodebefore_model
modify_model_request
before_model
calls.
These functions cannot modify permanent state or exit early.
Rather, they are intended to modify calls to the model in a **stateless* way.
If you want to modify calls to the model in a **stateful** way, you will need to use before_model
Modifies the model request. The model request has several key properties:
model
(BaseChatModel
): the model to use. Note: this needs to the base chat model, not a string.
system_prompt
(str
): the system prompt to use. Will get prepended to messages
messages
(list of messages): the message list. Should not include system prompt.
tool_choice
(Any): the tool choice to use
tools
(list of BaseTool
): the tools to use for this model call
responseFormat
(ResponseFormat
): the response format to use for structured output
after_model
stateSchema
, these properties must be provided when invoking the agent:
.default()
in Zod schemas to make properties optionalbeforeModel
: Are run in the order they are passed in. If an earlier middleware exits early, then following middleware are not run
modifyModelRequest
: Are run in the order they are passed in.
afterModel
: Are run in the reverse order that they are passed in. If an earlier middleware exits early, then following middleware are not run
jump_to
key to the state update with one of the following values:
"model"
: Jump to the model node"tools"
: Jump to the tools node"__end__"
: Jump to the end nodemodel
node, all beforeModel
middleware will run. It’s forbidden to jump to model
from an existing beforeModel
middleware.
Example usage: