You can log custom metadata keys when sending requests through the AI Gateway. This allows you to tag requests with additional context, which can be used for observability and request control. With metadata, you can:
  • Enhance Observability: Filter requests to get specific insights and create custom metrics pages for advanced monitoring.
  • Apply Conditional Configurations: Use metadata in the when block of your gateway configurations to selectively apply rules for rate limiting, model fallbacks, load balancing, and more.

​
How to Log Custom Metadata

The custom metadata is logged as key value pairs in the X-TFY-METADATA header. Here’s an example of how to log custom metadata: 
from openai import OpenAI

USE_STREAM = True
client = OpenAI(api_key="****", base_url="https://llm-gateway.truefoundry.com/api/inference/openai")
stream = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are an AI bot."},
        {"role": "user", "content": "Enter your prompt here"},
    ],
    model="openai-main/gpt-4",
    stream=USE_STREAM,
    extra_headers={
        "X-TFY-METADATA": '{"application":"booking-bot", "environment":"staging", "customer_id":"123456", "tfy_log_request":"true"}',
    }
)

if USE_STREAM:
    for chunk in stream:
        if (
            chunk.choices
            and len(chunk.choices) > 0
            and chunk.choices[0].delta.content is not None
        ):
            print(chunk.choices[0].delta.content, end="")
else:
    print(stream.choices[0].message.content)
You can include multiple metadata keys with each request. Please note that all values must be strings, with a maximum length of 128 characters.

​
Using Metadata for Observability

Metadata keys are instrumental for monitoring and debugging. You can use them to filter your request logs and build custom metric dashboards.

​
Filter Logs

Filter your logs using one or more metadata keys to quickly isolate specific requests. This is useful for debugging or analyzing usage patterns for a particular feature, environment, or user.

Filtering requests using custom Metadata Keys

​
Create Custom Metrics

You can also group metrics by metadata keys to create custom visualizations. For example, you can monitor cost and usage per customer by grouping with a customer_id key.

Grouping metrics by a custom 'customer_id' metadata key.

​
Using Metadata in Gateway Configurations

Metadata is powerful for creating conditional rules in your AI Gateway. You can use the metadata field inside a when block to apply configurations only when certain metadata is present in the request. Here are a few examples:

​
Rate limit

You can rate limit your requests based on metadata. For example, to limit all requests to the gpt-4 model from the dev environment to 1000 requests per day. 
name: ratelimiting-config
type: gateway-rate-limiting-config
rules:
  - id: 'openai-gpt4-dev-env'
    when:
      models: ['openai-main/gpt4']
      metadata:
        env: dev
    limit_to: 1000
    unit: requests_per_day
Similarly, you can use metadata to configure Load Balancing and Fallbacks. For example, you could direct traffic based on customer-id or set up specific fallback models for a feature. Learn more in the Load Balancing and Fallback documentation.