The LLM Gateway is in beta.
Understand fallback behavior
Chain components
A model fallback chain has:- A primary model: the provider and model that trigger the chain when a request fails.
- One to five fallbacks: an ordered list of provider models or saved model configurations. A provider model uses the workspace’s secret for that provider, or Gateway Credits for eligible hosted models.
- Fallback triggers: the HTTP status codes that move the request to the next model. The defaults cover rate-limit, overload, server, and request timeout errors, such as
408,429,500, and503. - A time to first output: how long each attempt can wait before the gateway moves to the next model. It is on by default at 120 seconds, and each attempt gets a fresh timer. Streaming requests in a unified API format wait for the first output. Non-streaming requests and direct model access requests wait for response headers.
- A model cooldown (optional): how many failures make the gateway skip a model, and for how long. It is off by default. See Model cooldowns to learn more.
Model cooldowns
A cooldown is a period during which the gateway skips a model in a fallback chain without calling it. Skipping a model that is likely to fail saves each request the latency of a failed attempt. A model enters a cooldown in two ways:- Provider rate limits: when a provider returns
429, the gateway skips that model for the retry delay in the provider’s response. If the response sets no delay, the cooldown lasts 5 seconds. No cooldown lasts longer than 300 seconds. Every chain in the workspace whose triggers include429skips the model. - Model cooldown: when you turn on Model cooldown for a chain, a model that fails the chain’s triggers a set number of times within the cooldown period is skipped by that chain for the cooldown period. Trigger statuses, timeouts, and provider transport errors count as failures. Other statuses and canceled requests do not, and a success does not reset the count. Failures are counted per model in each chain, so other chains keep calling the model. When on, a model is skipped for 60 seconds after 3 failures by default. When the period ends, the gateway calls the model again, and its failure count starts over.
Request flow
For each request:- The gateway matches the request’s provider-prefixed model ID to a fallback chain and calls the primary model.
- If the attempt fails with a trigger status, times out, or hits a transport error, the gateway calls the next fallback in order.
- The gateway repeats the previous step until one of the following:
- A model succeeds.
- A model returns a status that is not a trigger.
- No fallbacks remain.
- The gateway returns the last response in the API format used by the client.
400, which ends the chain unless 400 is a trigger status.
The gateway skips any model that is cooling down, except the last, without calling it. A skipped model does not appear in the response body. To see skipped models, investigate the chain’s attempts.
Once a streaming response starts sending output, the gateway commits to that model and no longer falls back, even if the stream fails later.
Supported formats and endpoints
Fallbacks can use a different provider and API format than the primary model. The gateway translates requests and responses between supported API formats. For example, an Anthropic primary can fall back to an OpenAI model without client-side changes. Fallback chains apply to Chat Completions, Responses, and Messages requests, on both the unified endpoints and direct model access paths. Other endpoints, such as embeddings and GeminigenerateContent requests through direct model access, call the requested model without a fallback.
Create a fallback chain
To create a fallback chain:- Go to LLM Gateway, and select the Model Fallbacks tab.
- Click Create fallback chain.
- Select the Workspace where the chain applies.
- Select the primary provider and model. Requests to this provider-prefixed model ID use the chain when the primary attempt fails.
- Under Fallbacks, add one to five backup models in the order the gateway should try them. Choose a provider and model directly, select an existing model configuration, or create a custom model configuration.
- Under Fallback triggers, review the status codes that send the request to the next model. Adjust them as needed.
- (Optional) Adjust Time to first output, in seconds. Leave the field blank to remove the timeout.
- (Optional) Turn on Model cooldown, and set how long to skip a model and how many failures start a cooldown.
- Click Create chain.
Test a fallback chain
The Simulator sends a request through a fallback chain with a behavior you choose for each model, such as a rate limit or an outage. This way, you can check the chain’s routing and added latency before a provider fails in production. To test a chain:- Go to LLM Gateway, and select the Model Fallbacks tab.
- On the chain’s row, click Open in Simulator →.
- Under Simulate provider behaviour, choose a behavior for each model in the chain (Healthy, Timeout, Rate limit, or Outage).
- Click Run request.
- Simulated failures send no inference request. To approximate network latency, the gateway calls a provider endpoint that does not generate tokens. Simulated failures appear as attempt runs in the request trace, without a provider LLM run.
- A simulated failure falls back only on a trigger status. Rate limit and Outage move to the next model only while
429and503are in the chain’s triggers. Both are in the defaults. - The Simulator sends non-streaming requests only. To check a chain’s streaming behavior, send a streaming request from your code.
- The Simulator ignores cooldowns. It calls every model even while one is cooling down, and its failures do not count toward Model cooldown, so a test run cannot cool down a model for production traffic. To check cooldown behavior, send requests through the chain from your code.
- Errors outside the simulation still surface. If a fallback has no provider key in the workspace, its attempt returns
424, as it would in production. If the primary has no key, the Simulator asks you to add one before it runs the chain.
Make a call
Call the standard LLM Gateway endpoint with the primary model’s provider-prefixed ID. You do not need a route-specific URL or additional request fields. This example uses the base URL and API key set in Send your first LLM Gateway request:<PRIMARY_MODEL_ID> in the API key’s workspace. If no chain matches, the gateway returns the primary model’s response without attempting a fallback.
Investigate fallback attempts
When a chain runs, the response records which models the gateway tried, each attempt’s status code, and the model that served the request:- In gateway traces: Open the request in the gateway tracing project, and expand
gateway.request. Eachgateway.attempt #Ncontains its policy checks and, when made, its provider LLM run. Select an attempt to inspect its outcome and upstream error metadata. - In application traces: If your application calls the gateway through a LangChain chat model in Python (
langchain-openai>=1.6.7orlangchain-anthropic>=1.7.5) with tracing on, the model’s run has a Gateway section that lists each attempt. The chat model reads this from the response header described in the next item. - In the response: other clients can read the
X-LangSmith-Gateway-Metadataresponse header. Itsattempt_detailsfield lists each attempt’s model and status code, andselected_modelnames the model that served the request. A model skipped because it was cooling down hasfallback_triggerset tocooldownand no status code.
Set fallbacks for a prompt
A prompt saved in LangSmith stores a model selection with its template, so your application can pull and run both together. A fallback chain applies to every request for its primary model in its workspace. To give one prompt fallback behavior that differs from other traffic to the same model, point the prompt at its own saved model configuration, and make that configuration the chain’s primary. The prompt calls the configuration ascustom/<name>, and the gateway applies the configuration’s chain.
To configure fallbacks for a prompt:
- In Settings > Model configurations, create a model configuration for the provider and model you want the prompt to use. Note the configuration’s name,
<config_name>, which the next two steps use. An organization-scoped configuration reads the provider key from organization secrets, so if the key is a workspace secret, create a workspace configuration. - Create a fallback chain in the same workspace. For the primary model, select your configuration from the Model configurations section at the end of the model list, rather than its underlying provider and model. The chain’s primary model is
custom/<config_name>, separate from any chain on the underlying model. - Create a prompt in the Playground. Click the current model’s name to open Model Configuration, and select LangSmith Gateway as the Provider. Type
custom/<config_name>in the Model field, and click Apply. Click Start to check that the prompt runs. Click Save, and name the prompt. - In your application, pull the prompt with its model. In Python, set
include_model=Truewhen callingclient.pull_prompt.
See also
- Add a provider key: give each model in a chain a provider key.
- Monitor LLM Gateway spend: review the cost of primary and fallback attempts by model.
- Spend policies: apply cost limits alongside fallback routing.
Connect these docs to your agent of choice via MCP for real-time answers.

