Fallback a request to another model on failure from one model
X-TFY-METADATA
. The subjects, models and metadata fields are conditioned in an AND fashion - meaning that the rule will only match if all the conditions are met. If an incoming request doesn’t match the when block in one rule, the next rule will be evaluated.
subjects
: Filter based on the list of users / teams / virtual accounts calling the model. User can be specified using user:john-doe
or team:engineering-team
or virtual-account:acct_1234567890
.models
: Rule matches if the model name in the request matches any of the models in the list.metadata
: Rule matches if the metadata in the request matches the metadata in the rule. For e.g. if we specify metadata: {environment: "production"}
, the rule will only match if the request has the metadata key environment
with value production
in the request header X-TFY-METADATA
.response_status_codes
: The response status on which the fallbacks will be executed. This is important since we only want to retry on recoverable errors codes like rate limit exceeded (429), etc.target
: The model ID to which traffic should be routedoverride_params
(optional): A key-value object used to modify or extend the request body when falling back to the target model. Each key corresponds to a parameter name, and the associated value specifies what should be sent to the target. This allows you to override existing parameters from the original request or introduce entirely new ones, enabling flexible customization for each target model during fallback.