Skip to main content
Beta: The LLM Gateway is in beta.
Model fallbacks retry a request against one or more backup model configurations when the primary model returns an error you’ve flagged as retryable, such as a rate limit or a provider outage. Instead of building retry logic into every agent, define the fallback order once in LangSmith and call it through a single gateway route.

​
How it works

A fallback configuration has:
  • A name: exposed at the route https://gateway.smith.langchain.com/routes/{name}. Clients call this URL instead of a provider-specific path.
  • One or more fallback chains: each an ordered list of model configurations to try in priority order.
  • Triggers: the upstream HTTP status codes that should cause the gateway to move on to the next model in the chain. For example, 429 for rate limits, or 500, 502, 503, 504 for other provider errors.
For each request, the gateway:
  1. Determines the wire format from the request path, and considers only the chains in that format (see Wire formats).
  2. Picks one of those chains (see Selecting a chain).
  3. Calls the chain’s first model configuration.
  4. If the response status matches a configured trigger, discards that response and calls the next model configuration in the chain.
  5. Repeats until a candidate returns a non-trigger response, or the chain is exhausted—in which case the last candidate’s response is returned to the caller.
Each configuration in a chain can point to a different provider and model. On both the first attempt and any fallback, the gateway calls the candidate with its configured model name, replacing the value the client sent. The client’s model field only selects which chain to use from those matching the path’s wire format. If no chain matches, the gateway uses the first chain defined for that wire format as the default.

​
Wire formats

Each chain is either OpenAI-compatible or Anthropic-compatible, since the gateway forwards the same request body to every candidate in it: Only these two provider types can go in a chain. A model configuration saved for another provider, such as Azure OpenAI or Bedrock, isn’t eligible. To reach a host that speaks the OpenAI API, save it as an OpenAI Compatible Endpoint with its base URL. The path you call selects the format, and the gateway only considers chains in that format. Any other path returns 501 Not Implemented. A single fallback configuration can hold chains of both wire formats, so one route can serve both OpenAI-compatible and Anthropic-compatible clients. Mixing formats is optional: a configuration with only OpenAI-compatible chains returns 502 for /v1/messages calls, and one with only Anthropic-compatible chains returns 502 for /v1/chat/completions calls.

​
Create a fallback configuration

Creating and managing fallback configurations requires organization:manage permission. For the full permissions breakdown, refer to access control.
  1. Go to Settings → Gateway → LLM Gateway and select the Model Fallbacks tab.
  2. Click Create configuration.
  3. Enter a Configuration name. This becomes {name} in the gateway URL https://gateway.smith.langchain.com/routes/{name}.
  4. Select the Workspace the configuration belongs to. Model configurations are workspace-scoped, so only that workspace’s configurations are available to add to a chain. The workspace can’t be changed later—delete and recreate the configuration to move it.
  5. Under Fallback triggers, review the HTTP status codes that should trigger a fallback. The list comes prepopulated with the transient codes another provider has a chance of serving (such as 429, 500, and 503); add or remove codes as needed.
  6. Under Model fallback chains, click Add chain, then add two to five model configurations in the order the gateway should try them. A chain’s first model fixes its wire format; only configurations of that format can follow. The gateway groups chains by format, and the first chain in each group is that format’s default, which it uses when a request’s model does not select another chain.
  7. Click Create configuration.

​
Make a call

Call the route the same way you’d call a custom provider, on the path for the wire format you want:
The gateway tries each model configuration in the selected chain, in order, until one responds without a trigger status. Each attempt is traced and counted against spend policies on its own, so a request that falls back records one call per candidate tried.

​
Selecting a chain

A configuration can hold more than one fallback chain, which is useful when different model families need different fallback behavior. Among the chains matching the request’s wire format, the gateway selects one by the request body’s model field, in this order:
  1. If model matches a chain’s alias, that chain is used.
  2. Otherwise, if model matches a chain’s primary (first) model configuration’s underlying model name, that chain is used.
  3. If model is omitted, or matches neither, the first (default) chain of that format is used.
An alias is optional and set per chain when you create the configuration. It is a caller-facing name, so a client can ask for "model": "heavy" without knowing which model backs it. Each model configuration in a chain can point to a different host, so a chain can fail over across separate deployments of the same model (for example, from a primary endpoint to a backup) with no single host as a point of failure. For example, a configuration with three chains:
  • A /v1/chat/completions request with "model": "gpt-5.5" uses chain 1, which fails over from OpenAI to Azure OpenAI to the self-hosted endpoint.
  • A /v1/chat/completions request with "model": "gpt-4o-mini" uses chain 2, which fails over from OpenAI to Fireworks.
  • A /v1/chat/completions request with an unrecognized or omitted model uses chain 1, the default for that format.
  • A /v1/messages request uses chain 3 whatever its model is, because it’s the only Anthropic-compatible chain. Chains 1 and 2 are never eligible on that path.

​
Next steps

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.
Edit this page on GitHub or file an issue.