Truefoundry AI Gateway provides a wide range of features to help you manage your LLM models and use them in your applications. Here are some of the key features:

Unified API Interface

Single API interface to access multiple LLM providers with unified endpoint

Rate Limiting

Control Models Usage with flexible rate limiting policies per user/model/application

Multimodal Inputs

Support for text, image, and audio inputs across compatible models

Fallback

Automatic failover to backup models when primary models are unavailable

Load Balancing

Distribute requests across multiple model instances for optimal performance

API Key Management

Generate and manage API keys for users/applications

Guardrails

Content filtering and safety checks to ensure

Observability & Metrics

Comprehensive monitoring, logging, and analytics for all API requests

Access Control

Fine-grained access control and permissions management

Custom Metadata Routing

Route requests based on custom metadata and business logic

Latency-Based Load Balancing

Intelligent routing based on real-time latency metrics

Prompt Management

Centralized prompt versioning and management system

Tracing

Distributed tracing for debugging and performance optimization

Responses API

Advanced response handling and transformation capabilities

Batch Predictions

Process multiple requests efficiently with batch processing

PII Detection

Automatically detect and filter personally identifiable information

Real-time API

Low-latency streaming responses for real-time applications

Export Logs & Traces

Export request logs and traces for external analysis and compliance

Self-Hosted Models

Deploy and manage your own custom models alongside public providers