How the gateway works

The LLM Gateway sits between your application and the model providers your workspace has configured. It authenticates the caller, selects an upstream route, applies governance policies, translates between API formats, and traces the result.

​

What the gateway provides

• One key, multiple providers: Developers authenticate with a LangSmith API key instead of storing provider keys locally.
• One request format, multiple models: Use Chat Completions, Messages, or Responses with models across configured providers.
• Built-in observability: Every gateway call appears in gateway tracing projects, with visibility controlled by Traces and access control.
• Central governance: Apply spend limits, rate limits, and data policies.

​

Follow a request through the gateway

The gateway performs these steps for each request to the standard endpoint:

1. Authenticate and authorize the caller from the LangSmith API key, including the gateway:invoke permission check.
2. Resolve the route from the endpoint and the model ID, including any configured fallback candidates.
3. Load the upstream credential, either a workspace Provider Secret or a Gateway Credits credential.
4. Evaluate pre-request policies. Spend, rate, and model-access limits can block the request. Data-protection policies run against the request body and redact it when configured.
5. Translate the request when the selected provider uses a different API format. For details, see Understand translation behavior.
6. Send the request upstream and receive or stream the provider response.
7. Translate the response back into the API format the client requested, and restore redaction placeholders where applicable.
8. Return the response to the caller, recording usage, cost, routing and policy metadata, and the LangSmith trace.

Direct model access skips steps 5 and 7. The gateway still authenticates, resolves credentials, evaluates policies, and traces the call, but it passes the request and response through in the provider’s native format.

​

Choose how credentials are managed

The gateway resolves an upstream credential for every call. A workspace can use its own provider accounts, Gateway Credits, or both:

​

Check availability

The gateway runs on LangSmith Cloud in every LangSmith region, and on BYOC, where it runs inside your data plane so that model requests and their traces stay in your VPC. Both use the same API formats, model IDs, policies, and tracing; only the hostname and path prefix differ.

​

Use a regional gateway

Replace gateway.smith.langchain.com with the hostname for your LangSmith region, and keep the same path for the API format you use:

​

Use a BYOC data plane

On BYOC, replace the gateway hostname with your data plane endpoint and prefix the path with /gateway: Authenticate with an API key scoped to a workspace in that data plane, passed either as an Authorization: Bearer token or as the provider API key. Provider secrets, model IDs, policies, and tracing behave the same as on Cloud.

​

See also

• Quickstart: make your first request, view its trace, and set a spend limit.
• Admin setup: enable the gateway, add provider credentials, and grant developer access.
• API formats: use Chat Completions, Messages, or Responses through the standard endpoint.
• Traces, Engine, and access control: see where gateway traces appear and who can view them.

---