Skip to main content
Private beta: The LLM Gateway is in private beta. Sign up for the waitlist to get access.
A rate limit policy restricts how many requests or tokens a subject can consume through the LLM Gateway in a short rolling time window. The gateway enforces the limit in real time and blocks any request that would push the subject past it, returning a 429 response with a Retry-After header:
The Retry-After value is the number of seconds until the current window resets. Clients should honor this header and back off before retrying. Rate limit policies and spend cap policies are complementary and can be applied together—spend caps for cost control, rate limits for throughput and traffic control.

​
Policy dimensions

Rate limit policies are evaluated for every incoming request. You can set a policy as a default (applying a blanket rate limit to all users, workspaces, or API keys or as a granular policy (an individual limit or a limit on a group of subjects).

​
Defaults vs. granular policies

Rate limit policies have two modes:
  1. Default policies apply automatically to every member of a subject dimension. Example: “Every user in this workspace gets a default cap of 100 requests per minute.” No need to create a policy per person.
  2. Granular policies target a named subject and override the default for that subject only. Example: “The on-call engineer gets 500 requests per minute.” Editing the default updates everyone still on it.

​
Independent enforcement

Each subject is tracked and enforced separately. One user hitting their limit does not affect other users.

​
Limits

A single rate limit policy can enforce multiple limits at once. For example, one policy can enforce both 100 requests per minute and 1,000,000 tokens per hour simultaneously. Each limit has three fields: Rules:
  • At least one limit is required per policy.
  • You cannot have two limits with the same metric and window combination within one policy.

​
Create a rate limit policy

Creating and managing policies requires organization:manage permission. For the full permissions breakdown, refer to Traces, Engine, and access control.
  1. Go to Settings → Gateway → LLM Gateway.
  2. Click Create policy.
  3. Select Rate limit as the policy type.
  4. Select the subject scope (user, workspace, or API key).
  5. Add one or more limits, each with a metric, window, and value.
  6. Save.
Policies take effect immediately.

​
Next steps

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.
Edit this page on GitHub or file an issue.