Model fallbacks

​

How it works

• A primary model: the provider and model that trigger the chain when a request fails.
• One to five fallbacks: an ordered list of provider models or saved model configurations. A provider model uses the workspace’s secret for that provider, or Gateway Credits for eligible hosted models.
• Triggers: the HTTP status codes that move the request to the next model. The defaults cover common timeout, rate-limit, and server errors, such as 408, 429, 500, and 503. They do not include 424, which the gateway returns when a model has no provider key in the workspace. Add 424 to let the chain skip a fallback without a key. A primary model without a key returns 424 without trying the chain.
• A first-output timeout: how long each attempt can wait before the gateway moves to the next model. It is on by default at 120 seconds, and each attempt gets a fresh timer. Streaming requests in a unified API format wait for the first output. Non-streaming requests and direct model access requests wait for response headers.

1. The gateway calls the primary model selected by the request’s provider-prefixed model ID.
2. If the request fails with a configured trigger status, times out, or hits a transport error, the gateway loads the matching fallback chain.
3. The gateway calls each fallback in order until one succeeds, returns a status that does not trigger another fallback, or the chain is exhausted.
4. The gateway returns the final response in the API format used by the client.

​

Create a fallback chain

1. Go to LLM Gateway, and select the Model Fallbacks tab.
2. Click Create fallback chain.
3. Select the Workspace where the chain applies.
4. Select the primary provider and model. Requests to this provider-prefixed model ID use the chain when the primary attempt fails.
5. Under Fallbacks, add one to five backup models in the order the gateway should try them. Choose a provider and model directly, select an existing model configuration, or create a custom model configuration.
6. Under Configure fallback triggers (advanced), review the HTTP status codes that trigger the next fallback. To add a status code, type it in the field, and select it from the list.
7. (Optional) In the same section, adjust the First output timeout per model (seconds), or turn off Time to first output to remove the timeout.
8. Click Create chain.

​

Test a chain in the Simulator

1. Go to LLM Gateway, and select the Model Fallbacks tab.
2. On the chain’s row, click Open in Simulator →.
3. Under Simulate provider behaviour, choose a behavior for each model in the chain (Healthy, Timeout, Rate limit, or Outage).
4. Click Run request.

• Simulated failures send no inference request. To approximate network latency, the gateway calls a provider endpoint that does not generate tokens. Simulated failures have no trace.
• A simulated failure falls back only on a trigger status. Rate limit and Outage move to the next model only while 429 and 503 are in the chain’s triggers. Both are in the defaults.
• The Simulator sends non-streaming requests only. To check a chain’s streaming behavior, send a streaming request from your code.
• Errors outside the simulation still surface. If a fallback has no provider key in the workspace, its attempt returns 424, as it would in production. If the primary has no key, the Simulator asks you to add one before it runs the chain.

​

Make a call

curl "$LANGSMITH_GATEWAY_BASE_URL/chat/completions" \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"<PRIMARY_MODEL_ID>","messages":[{"role":"user","content":"Hello!"}]}'

​

Investigate fallback attempts

• In your traces: if your application calls the gateway through a LangChain chat model in Python (langchain-openai>=1.6.7 or langchain-anthropic>=1.7.5) with tracing on, the model’s run has a Gateway section that lists each attempt.
• In the response: other clients can read the X-LangSmith-Gateway-Metadata response header. Its attempt_details field lists each attempt’s model and status code, and selected_model names the model that served the request.

​

Set fallbacks for a prompt

1. In Settings > Model configurations, create a model configuration for the provider and model you want the prompt to use. Note the configuration’s name, <config_name>, which the next two steps use. An organization-scoped configuration reads the provider key from organization secrets, so if the key is a workspace secret, create a workspace configuration.
2. Create a fallback chain in the same workspace. For the primary model, select your configuration from the Model configurations section at the end of the model list, rather than its underlying provider and model. The chain’s primary model is custom/<config_name>, separate from any chain on the underlying model.
3. Create a prompt in the Playground. Click the current model’s name to open Model Configuration, and select LangSmith Gateway as the Provider. Type custom/<config_name> in the Model field, and click Apply. Click Start to check that the prompt runs. Click Save, and name the prompt.
4. In your application, pull the prompt with its model. In Python, set include_model=True when calling client.pull_prompt.

​

See also

• Add a provider key: give each model in a chain a provider key.
• Monitor LLM Gateway spend: review the cost of primary and fallback attempts by model.
• Spend policies: apply cost limits alongside fallback routing.