Skip to main content
The LLM Gateway is in beta.
A rate limit policy restricts how many requests or tokens a subject can consume through the LLM Gateway in a short rolling time window. The gateway enforces the limit in real time and blocks any request that would push the subject past it, returning a 429 response with a Retry-After header and a body that names the blocking policy:
The Retry-After value is the number of seconds until the current window resets. Clients should honor this header and back off before retrying. A blocked request never reaches the provider, so it does not trigger a fallback chain, even though 429 is a default fallback trigger.
Rate limit policies and spend cap policies are complementary and can be applied together: spend caps for cost control, rate limits for throughput and traffic control.

​
Policy dimensions

Rate limit policies are evaluated for every incoming request.

​
Defaults vs. granular policies

Rate limit policies have two modes:
  1. Default policies apply automatically to every member of a subject dimension. Example: “Every user in this workspace gets a default cap of 100 requests per minute.” There is no need to create a policy per person.
  2. Granular policies target a named subject and override the default for that subject only. Example: “The on-call engineer gets 500 requests per minute.” Editing the default updates everyone still on it.

​
Independent enforcement

Each subject is tracked and enforced separately. One user hitting their limit does not affect other users.

​
Limits

A single rate limit policy can enforce multiple limits at once. For example, one policy can enforce both 100 requests per minute and 1,000,000 tokens per hour simultaneously. Each limit has three fields: Rules:
  • At least one limit is required per policy.
  • You cannot have two limits with the same metric and window combination within one policy.

​
Create a rate limit policy

Creating and managing policies requires organization:gateway:manage permission. Organization Admins and Organization Operators have it by default. For what each organization role can do, see Organization roles.
  1. Go to LLM Gateway and select Rate limiting.
  2. Click Create rate limit.
  3. Select the scope (organization, workspace, user, or API key).
  4. Add one or more limits, each with a value, metric, and window.
  5. Click Create rate limit.
Policies take effect immediately. A rate limit policy can also carry a condition on a custom request header, so traffic from a single subject splits into separate limits by header value. Use this to give each end customer, tenant, or team its own throughput allowance under one API key. For more information, see Header-based policies.

​
See also

Connect these docs to your agent of choice via MCP for real-time answers.
Edit this page on GitHub or file an issue.