Introduction
TrueFoundry AI Gateway is the proxy layer that sits between your applications and the LLM providers and MCP Servers. It is an enterprise-grade platform that enables users to access 1000+ LLMs using a unified interface while taking care of observability and governance.
Key Features
Unified API Interface
Call 1000+ LLMs using a single endpoint with unified API interface
API Keys Management
Generate and manage API keys for users/applications
Multimodal Inputs
Support for text, image, and audio inputs across compatible models
Access Control
Fine-grained access control and permissions management
Rate Control
Control Models Usage with flexible rate limiting policies per user/model/application
Fallback
Automatic failover to backup models when primary models are unavailable
Load Balancing
Distribute requests across multiple model instances based on weight, latency or cost metrics.
Budget Limiting
Control spending and enforce cost limits for users, teams, and models
Guardrails
Content filtering and safety checks to ensure
Observability & Metrics
Opentelemetry compliant metrics and logging for all requests.
Prompt Playground
Centralized prompt playground with versioning and management system
Batch Predictions
Process multiple requests efficiently with batch processing
Integrate MCP Servers
Deploy and manage your own MCP servers with TrueFoundry AI Gateway.
Access Control for MCP Servers
Define which developers / teams / applications can access which MCP servers.
Centralized Authn/Authz for all MCP Servers
One API key to access all MCP servers and their tools.
Agent Playground
Test Agents by adding tools and models from Playground
Build Agents with unified API for all MCP servers
Connect to MCP Servers with a single API in the gateway.
Rate Limiting and Observability for Tools
Coming Soon
Why Choose TrueFoundry AI Gateway?
🔄 Zero Vendor Lock-In
🔄 Zero Vendor Lock-In
Our gateway is OpenAI compatible and can be used with any OpenAI-compatible client. User doesn’t need to change their existing codebase or clients to use truefoundry’s AI gateway.
Truefoundry doesn’t provide a client side SDK for the gateway and instead recommend using OpenAI client or requests library. This ensures that your code doesn’t get vendor locked in to Truefoundry and you can switch Gateways easily.
🔒 Enterprise-Grade Security
🔒 Enterprise-Grade Security
- Centralized API key management
- Granular access controls
- Guardrails (pii detection, content filtering, etc.)
- Complete audit trails
🏅 Trusted by Fortune 500 Companies: Multiple Fortune 500 companies are using TrueFoundry. Read about our case studies here.
📊 Observability
📊 Observability
- Real-time usage analytics per user, model, and provider
- Monitor latency, error rates, token usage, etc.
- Cost Tracking and Optimization
🛡️ Built for Scale
🛡️ Built for Scale
- Minimal latency overhead (less than 10ms)
- Edge deployment capabilities
- Rate limiting and abuse prevention
- High availability architecture
🚀 Performance Benchmark: Learn how TrueFoundry AI Gateway delivers blazing-fast performance
Read our benchmark analysis →
Supported Integrations
We integrate with 1000+ LLMs through the following providers.
If you don’t see the provider you need, there is a high change it will just work as self hosted models or OpenAI provider. Please reach out to us at support@truefoundry.com and we will be happy to guide you.
Deployment Options
The Truefoundry AI Gateway can either be used as a SaaS offering or deployed on-premise.
- SaaS Offering: You can directly use the gateway as a SaaS offering by signing up on our website, you can find the instructions here.
- Enterprise Deployment for enterprise security and control. You can deploy the gateway in your cloud or on-premise. You can find the architecture and deployment instructions here.
Frequently Asked Questions
What's the performance impact of using the gateway?
What's the performance impact of using the gateway?
The latency overhead is minimal, typically less than 5ms. Our benchmarks show enterprise-grade performance that scales with your needs.
Our SaaS offering is hosted in multiple regions across the world to ensure low latency and high availability. You can also deploy the gateway on-premise or on any cloud provider in your region which
is closer to your users.
AI Gateway on the edge, close to your applications for optimal performance
Can I deploy the gateway on-premise?
Can I deploy the gateway on-premise?
Yes, the AI Gateway supports on-premise deployments on any infrastructure or cloud provider, giving you complete control over your AI operations.
How do I integrate my self-hosted models?
How do I integrate my self-hosted models?
You can easily integrate any OpenAI-compatible self-hosted model. Check our self-hosted models guide for detailed instructions.
Can I use the gateway without the full MLOps platform?
Can I use the gateway without the full MLOps platform?
Yes, The AI Gateway can be used as a standalone solution. You can use the full MLOps platform if you’re using features like model deployment(traditional models and LLMs), model training, llm fine-tuning or training/data-processing workflows.
Introduction
TrueFoundry AI Gateway is the proxy layer that sits between your applications and the LLM providers and MCP Servers. It is an enterprise-grade platform that enables users to access 1000+ LLMs using a unified interface while taking care of observability and governance.
Key Features
Unified API Interface
Call 1000+ LLMs using a single endpoint with unified API interface
API Keys Management
Generate and manage API keys for users/applications
Multimodal Inputs
Support for text, image, and audio inputs across compatible models
Access Control
Fine-grained access control and permissions management
Rate Control
Control Models Usage with flexible rate limiting policies per user/model/application
Fallback
Automatic failover to backup models when primary models are unavailable
Load Balancing
Distribute requests across multiple model instances based on weight, latency or cost metrics.
Budget Limiting
Control spending and enforce cost limits for users, teams, and models
Guardrails
Content filtering and safety checks to ensure
Observability & Metrics
Opentelemetry compliant metrics and logging for all requests.
Prompt Playground
Centralized prompt playground with versioning and management system
Batch Predictions
Process multiple requests efficiently with batch processing
Integrate MCP Servers
Deploy and manage your own MCP servers with TrueFoundry AI Gateway.
Access Control for MCP Servers
Define which developers / teams / applications can access which MCP servers.
Centralized Authn/Authz for all MCP Servers
One API key to access all MCP servers and their tools.
Agent Playground
Test Agents by adding tools and models from Playground
Build Agents with unified API for all MCP servers
Connect to MCP Servers with a single API in the gateway.
Rate Limiting and Observability for Tools
Coming Soon
Why Choose TrueFoundry AI Gateway?
🔄 Zero Vendor Lock-In
🔄 Zero Vendor Lock-In
Our gateway is OpenAI compatible and can be used with any OpenAI-compatible client. User doesn’t need to change their existing codebase or clients to use truefoundry’s AI gateway.
Truefoundry doesn’t provide a client side SDK for the gateway and instead recommend using OpenAI client or requests library. This ensures that your code doesn’t get vendor locked in to Truefoundry and you can switch Gateways easily.
🔒 Enterprise-Grade Security
🔒 Enterprise-Grade Security
- Centralized API key management
- Granular access controls
- Guardrails (pii detection, content filtering, etc.)
- Complete audit trails
🏅 Trusted by Fortune 500 Companies: Multiple Fortune 500 companies are using TrueFoundry. Read about our case studies here.
📊 Observability
📊 Observability
- Real-time usage analytics per user, model, and provider
- Monitor latency, error rates, token usage, etc.
- Cost Tracking and Optimization
🛡️ Built for Scale
🛡️ Built for Scale
- Minimal latency overhead (less than 10ms)
- Edge deployment capabilities
- Rate limiting and abuse prevention
- High availability architecture
🚀 Performance Benchmark: Learn how TrueFoundry AI Gateway delivers blazing-fast performance
Read our benchmark analysis →
Supported Integrations
We integrate with 1000+ LLMs through the following providers.
If you don’t see the provider you need, there is a high change it will just work as self hosted models or OpenAI provider. Please reach out to us at support@truefoundry.com and we will be happy to guide you.
Deployment Options
The Truefoundry AI Gateway can either be used as a SaaS offering or deployed on-premise.
- SaaS Offering: You can directly use the gateway as a SaaS offering by signing up on our website, you can find the instructions here.
- Enterprise Deployment for enterprise security and control. You can deploy the gateway in your cloud or on-premise. You can find the architecture and deployment instructions here.
Frequently Asked Questions
What's the performance impact of using the gateway?
What's the performance impact of using the gateway?
The latency overhead is minimal, typically less than 5ms. Our benchmarks show enterprise-grade performance that scales with your needs.
Our SaaS offering is hosted in multiple regions across the world to ensure low latency and high availability. You can also deploy the gateway on-premise or on any cloud provider in your region which
is closer to your users.
AI Gateway on the edge, close to your applications for optimal performance
Can I deploy the gateway on-premise?
Can I deploy the gateway on-premise?
Yes, the AI Gateway supports on-premise deployments on any infrastructure or cloud provider, giving you complete control over your AI operations.
How do I integrate my self-hosted models?
How do I integrate my self-hosted models?
You can easily integrate any OpenAI-compatible self-hosted model. Check our self-hosted models guide for detailed instructions.
Can I use the gateway without the full MLOps platform?
Can I use the gateway without the full MLOps platform?
Yes, The AI Gateway can be used as a standalone solution. You can use the full MLOps platform if you’re using features like model deployment(traditional models and LLMs), model training, llm fine-tuning or training/data-processing workflows.