Skip to main content
The LLM Gateway is in beta.
The LLM Gateway sits between your application and the model providers your workspace has configured. It authenticates the caller, selects an upstream route, applies governance policies, translates between API formats, and traces the result.

Follow a request through the gateway

The gateway performs these steps for each request to the standard endpoint. The provider keys, permissions, and policies it applies come from setting up the gateway.
  1. Authenticate and authorize the caller from the LangSmith API key, including the gateway:invoke permission check. For details, see Authenticate to the LLM Gateway.
  2. Resolve the route from the endpoint and the model ID, including any configured fallback candidates.
  3. Load the upstream credential, such as a workspace provider key or a Gateway Credits credential. For details, see Provider authentication.
  4. Evaluate pre-request policies. Spend, rate, and model access limits can block the request. Data protection policies run against the request body and redact it when configured.
  5. Translate the request when the selected provider uses a different API format. For details, see Understand translation behavior.
  6. Send the request upstream and receive or stream the provider response.
  7. Translate the response back into the API format the client requested, and restore redaction placeholders where applicable.
  8. Return the response to the caller, recording usage, cost, routing and policy metadata, and the LangSmith trace. For details, see View LLM Gateway traces.
Direct model access skips steps 5 and 7. The gateway still authenticates, resolves credentials, evaluates policies, and traces the call, but it passes the request and response through in the provider’s native format.

Choose how credentials are managed

The gateway resolves an upstream credential for every call. A workspace can use its own provider keys, Gateway Credits, or both. For every credential source and who bills it, see Provider authentication.

Understand workspace scope

A request uses the settings of the workspace that its API key belongs to. Gateway settings differ in scope:
  • Provider keys: workspace secrets. Each workspace that calls a provider needs its own key.
  • Fallback chains: created in one workspace, and they apply to requests from that workspace’s API keys. See Model fallbacks.
  • Traces: recorded in the workspace of the API key that made the request.
  • Model configurations: workspace or organization scope. Organization-scoped configurations read organization secrets. See Custom model configurations.
  • Policies: each spend limit, rate limit, model access policy, and data policy applies to the whole organization, one workspace, one user, or one API key.

Check availability

The gateway runs on: All three use the same API formats, model IDs, policies, and tracing. Only the hostname and path prefix differ. On self-hosted LangSmith earlier than application version 0.18.9 (Helm chart versions before 0.18.0-rc.22), managing policies requires organization:manage instead of organization:gateway:manage.

Use a regional gateway

Replace gateway.smith.langchain.com with the hostname for your LangSmith region, and keep the same path for the API format you use:
API keys from one region do not work on another region’s gateway.

Use a BYOC data plane

On BYOC, replace the gateway hostname with your data plane endpoint and prefix the path with /gateway: Authenticate with an API key scoped to a workspace in that data plane, sent in any header listed in Send the key. Provider secrets, model IDs, policies, and tracing behave the same as on Cloud.
Data planes are provisioned with a private endpoint by default, so you need private connectivity to reach the base URL, such as Tailscale, AWS PrivateLink, or VPC peering.

Use a self-hosted installation

On self-hosted LangSmith, an administrator first enables the gateway in the Helm chart. Then replace the gateway hostname with your LangSmith hostname and prefix the path with /gateway: Authenticate with an API key scoped to a workspace in your installation, sent in any header listed in Send the key.

See also