Configure Ratelimits

TrueFoundry allows you to setup rate limit for specific users, teams and virtual accounts.

Setup rate limit for users

Say you want to limit all requests to gpt4 model from openai-main account for users [email protected] and [email protected] to 1000 requests per day

name: ratelimiting-config
type: gateway-rate-limiting-config
# The rules are evaluated in order, and all matching rules are considered.
# If any one of them causes a rate limit, the corresponding ID will be returned.
rules:
  # Limit all requests to gpt4 model from openai-main account for user:[email protected] and user:[email protected] to 1000 requests per day
  - id: "openai-gpt4-dev-env"
    when: 
      subjects: ["user:[email protected]", "user:[email protected]"]
      models: ["openai-main/gpt4"]
    limit_to: 1000
    unit: requests_per_day

Setup rate limit for teams

Say you want to limit all requests for team frontend to 5000 requests per day

name: ratelimiting-config
type: gateway-rate-limiting-config
# The rules are evaluated in order, and all matching rules are considered.
# If any one of them causes a rate limit, the corresponding ID will be returned.
rules:
  # Limit all requests for team frontend to 5000 requests per day
  - id: "openai-gpt4-dev-env"
    when: 
      subjects: ["team:frontend"]
    limit_to: 5000
    unit: requests_per_day

Setup rate limit for virtual accounts

Say you want to limit all requests for virtual account va-james to 1500 requests per day

name: ratelimiting-config
type: gateway-rate-limiting-config
# The rules are evaluated in order, and all matching rules are considered.
# If any one of them causes a rate limit, the corresponding ID will be returned.
rules:
  # Limit all requests for virtual account va-james to 1500 requests per day
  - id: "openai-gpt4-dev-env"
    when: 
      subjects: ["virtualaccount:va-james"]
    limit_to: 1500
    unit: requests_per_day

Learn more about rate limits here