Load Balancing
Overview
Optimize LLM performance and reliability through intelligent request distribution
TrueFoundry AI Gateway provides a load balancing feature that allows you to distribute requests across multiple models. Here are a few common scenarios when organizations need load balancing while working with LLMs in production.
To avoid the downtime of your applications when models go down, a lot of organizations use multiple model providers and configure load balancing to route to the healthy model in case one of the models goes down, hence avoiding any downtime of their applications for their users.
2. Latency Variance among models: Latency and performance varies on time, region, model and provider. Here’s a graph of the latency variance of a few models over a course of a month.
We want to be able to route dynamically to the model with the lowest latency at any point in time.
3. Rate Limits of Models: A lot of the LLM providers enforce strict rate limits on API usage. Here’s a screenshot of Azure OpenAI’s rate limits:
When these limits are exceeded, requests begin to fail and we want to be able to route to other models to keep our application running.
4. Canary Testing: Testing new models or updates in production carries significant risks. Dynamic load balancing can be used to route a small percentage of traffic to the new model and monitor the performance before routing all the traffic to the new model.
You can also store the configuration in YAML file in your Git repository and apply it to the Gateway using the
Why do we need load balancing?
1. Service Outages and Downtime: Model providers experience outages and downtimes. For e.g. here’s a screenshot of OpenAI’s and Anthropic’s status page from Feb to May 2025.OpenAI Status Page
Anthropic Status Page
Latency Variance of models over a course of a month
Azure OpenAI Rate Limits
How TrueFoundry AI Gateway solves these challenges
TrueFoundry AI gateway enables us to configure load-balancing rules to enable routing to the most suitable model as defined by the user. It can automatically route to the model with the lowest latency, to the model with the least failures, or based on priority levels. This can help provide a much higher availability and performance for your application.How is loadbalancing done in TrueFoundry AI Gateway?
The gateway keeps track of the requests / minute and the tokens / minute for each model configured in the gateway. It also keeps track of the failures / minute for each model. Based on these metrics and the values configured by the user in the load balancing configuration, the gateway will first evaluate the list of healthy models to which a request can be routed. Among the list of healthy models, the gateway will then choose the model based on the load balancing strategy configured by the user - which can be weight-based, latency-based, or priority-based.Weight-based Routing
In weight-based routing, user assigns a weight to each target model. The gateway distributes incoming requests to the models in proportion to their assigned weights.- For example, if a request for the model
gpt-4ois received:- You can configure 90% of the requests to go to
azure/gpt-4oand 10% toopenai/gpt-4o. - The gateway will automatically route requests according to these specified ratios.
- You can configure 90% of the requests to go to
Latency-based Routing
In case of latency-based routing, the user doesn’t need to define the weights for each of the target models. The gateway will automatically choose the model with the lowest latency. The lowest latency model is chosen based on the following algorithm:- The time per output_token is used as a latency metric to evaluate the latency of the models.
- To calculate latency, only the requests in the last 20 mins are considered. If the count of requests in the last 20 mins is greater than 100, the last 100 requests are considered. If the count of requests in the last 20 mins is less than 100, all the requests are considered. If there are less than 3 requests in the last 20 mins, the latency is not derived and the model is considered as the fastest model so that more requests can be routed to it to get more data to derive the latency.
- Models are considered equally fast if their latency is within 1.2X of the fastest model - this is done to avoid rapid switching of models due to minor difference in latency.
Priority-based Routing
In priority-based routing, the user defines a priority level for each target model. The gateway will route requests to the highest priority model (lowest priority number) that is healthy and available. If the highest priority model fails or becomes unavailable, the gateway will automatically fallback to the next highest priority model. Lower priority numbers indicate higher priority (0 is the highest priority).Retry and Fallback Mechanisms
The gateway supports configurable retry mechanisms for each target:- Retry Configuration: Define the number of attempts, delay between retries, and status codes that trigger retries
- Fallback Configuration: Define status codes that trigger fallback to other targets and specify which targets can be used as fallback candidates
Configure Load Balancing on Gateway
The configuration can be added through the Config tab in the Gateway interface.Load Balancing Configuration Interface
tfy apply command. This enables enforcing a PR review process for any changes in the load balancing configuration.
Commonly Used Load Balancing Configurations
Here are a few examples of load balancing configurations for different use cases.Fallback to next model when rate limited
Fallback to next model when rate limited
Use priority-based routing and only move to next model when rate limited. This is useful when you have a preferred model (e.g., due to quality, compliance, or cost) and only want to use alternatives when the primary is rate limited (HTTP 429).
YAML
Canary Deployment
Canary Deployment
Gradually roll out a new gpt-4 model version. This is useful when you want to test a new model version before fully rolling it out to all users.
YAML
Priority-based Failover
Priority-based Failover
Route traffic based on priority with automatic failover. This is useful when you want to ensure that responses are always available and that the highest priority model is used.
YAML
Latency-based Performance Optimization
Latency-based Performance Optimization
Route traffic based on latency with retry and fallback. This is useful when you want to ensure that the response is as fast as possible.
YAML
Environment Based Routing
Environment Based Routing
Route traffic dynamically based on the deployment environment (development or production).
- In the development environment, all traffic is routed to a single model for simplicity and cost efficiency.
- In the production environment, traffic is intelligently distributed across multiple models using advanced load balancing strategies (e.g., latency-based or weight-based), ensuring high availability and optimal performance.
YAML
User-Proximity Routing
User-Proximity Routing
Optimize routing for geographically diverse users to minimize network latency. This is useful when you want to ensure that the response is as fast as possible for users in different regions.
- In the APAC region, traffic is routed to the model in the Southeast Asia region.
- In the US region, traffic is routed to the model in the East US region.
YAML
Burst Spillover to Fastest Healthy Target
Burst Spillover to Fastest Healthy Target
Automatically route traffic away from hotspots to the fastest available target. This is useful when you want to ensure that the response is as fast as possible for users in different regions.
YAML
Cost-based Weighted Routing
Cost-based Weighted Routing
Route traffic based on cost optimization, assigning higher weights to cheaper models. This is useful when you want to ensure that the response is as fast as possible for users in different regions.
YAML
Advanced Configuration with fallback & retry configuration
Advanced Configuration with fallback & retry configuration
This example demonstrates an advanced and production-ready load balancing configuration for the TrueFoundry AI Gateway, showcasing how to combine weight-based routing with granular per-target retry and fallback controls.Key features of this configuration:
- Rule Scope: The rule is scoped to requests originating from the
engineering-teamonly, ensuring targeted traffic management. - Weighted Traffic Distribution: Requests are distributed across three Anthropic Claude models using explicit weights (70%, 20%, 10%). The sum of weights must be 100. If weights are not specified, the configuration will be invalid.
- Per-Target Fallback Eligibility: You can control which targets are eligible to receive fallback traffic by setting
fallback_candidate(defaults totrueif not specified). - Custom Fallback Status Codes: For each target, you can specify which HTTP status codes should trigger a fallback to another eligible target using
fallback_status_codes. If not set, the default is["401", "403", "404", "429", "500", "502", "503"]. - Per-Target Retry Policy: Each target can optionally specify a custom retry policy via
retry_config. Ifretry_configis not provided for a target, the following defaults are automatically applied:attempts: 2delay: 100 (milliseconds)on_status_codes:["429", "500", "502", "503"]This ensures robust retry behavior out of the box, while still allowing you to override these settings per target as needed.
- Fine-Grained Reliability Control: This setup allows you to:
- Route the majority of traffic to the most reliable or cost-effective model.
- Exclude certain targets from fallback (e.g., for cost or compliance reasons).
- Tune retry and fallback behavior per target for optimal reliability and cost control.
- Target 1: Receives 70% of traffic, retries failed requests up to 2 times with a 150ms delay on specific status codes, and is eligible for fallback.
- Target 2: Receives 20% of traffic, does not retry failed requests, and is not eligible for fallback.
- Target 3: Receives 10% of traffic, retries up to 2 times with a 100ms delay, is eligible for fallback, and specifies custom fallback status codes.
YAML