The LLM Gateway is in beta.
Follow a request through the gateway
The gateway performs these steps for each request to the standard endpoint. The provider keys, permissions, and policies it applies come from setting up the gateway.- Authenticate and authorize the caller from the LangSmith API key, including the
gateway:invokepermission check. For details, see Authenticate to the LLM Gateway. - Resolve the route from the endpoint and the model ID, including any configured fallback candidates.
- Load the upstream credential, such as a workspace provider key or a Gateway Credits credential. For details, see Provider authentication.
- Evaluate pre-request policies. Spend, rate, and model access limits can block the request. Data protection policies run against the request body and redact it when configured.
- Translate the request when the selected provider uses a different API format. For details, see Understand translation behavior.
- Send the request upstream and receive or stream the provider response.
- Translate the response back into the API format the client requested, and restore redaction placeholders where applicable.
- Return the response to the caller, recording usage, cost, routing and policy metadata, and the LangSmith trace. For details, see View LLM Gateway traces.
Choose how credentials are managed
The gateway resolves an upstream credential for every call. A workspace can use its own provider keys, Gateway Credits, or both. For every credential source and who bills it, see Provider authentication.Understand workspace scope
A request uses the settings of the workspace that its API key belongs to. Gateway settings differ in scope:- Provider keys: workspace secrets. Each workspace that calls a provider needs its own key.
- Fallback chains: created in one workspace, and they apply to requests from that workspace’s API keys. See Model fallbacks.
- Traces: recorded in the workspace of the API key that made the request.
- Model configurations: workspace or organization scope. Organization-scoped configurations read organization secrets. See Custom model configurations.
- Policies: each spend limit, rate limit, model access policy, and data policy applies to the whole organization, one workspace, one user, or one API key.
Check availability
The gateway runs on:- LangSmith Cloud in every LangSmith region
- Bring your own cloud (BYOC), where it runs inside your data plane so that model requests and their traces stay in your VPC
- Self-hosted LangSmith
organization:manage instead of organization:gateway:manage.
Use a regional gateway
Replacegateway.smith.langchain.com with the hostname for your LangSmith region, and keep the same path for the API format you use:
API keys from one region do not work on another region’s gateway.
Use a BYOC data plane
On BYOC, replace the gateway hostname with your data plane endpoint and prefix the path with/gateway:
Authenticate with an API key scoped to a workspace in that data plane, sent in any header listed in Send the key. Provider secrets, model IDs, policies, and tracing behave the same as on Cloud.
Use a self-hosted installation
On self-hosted LangSmith, an administrator first enables the gateway in the Helm chart. Then replace the gateway hostname with your LangSmith hostname and prefix the path with/gateway:
Authenticate with an API key scoped to a workspace in your installation, sent in any header listed in Send the key.
See also
- Send your first LLM Gateway request: make your first request and view its trace.
- Set up the LLM Gateway: add provider keys, grant developer access, and set policies.
- API formats: use Chat Completions, Messages, or Responses through the standard endpoint.
- View LLM Gateway traces: see where gateway traces appear and who can view them.
Connect these docs to your agent of choice via MCP for real-time answers.

