Skip to main content
TrueFoundry AI Gateway is the proxy layer that sits between your applications and the LLM providers and MCP Servers. It is an enterprise-grade platform that enables users to access 1000+ LLMs using a unified interface while taking care of observability and governance. TrueFoundry AI Gateway architecture diagram showing the gateway as a proxy between applications and multiple LLM providers

Key Features

Why Choose TrueFoundry AI Gateway?

Our gateway is OpenAI compatible and can be used with any OpenAI-compatible client. User doesn’t need to change their existing codebase or clients to use truefoundry’s AI gateway.
Truefoundry doesn’t provide a client side SDK for the gateway and instead recommend using OpenAI client or requests library. This ensures that your code doesn’t get vendor locked in to Truefoundry and you can switch Gateways easily.
  • Centralized API key management
  • Granular access controls
  • Guardrails (pii detection, content filtering, etc.)
  • Complete audit trails
🏅 Trusted by Fortune 500 Companies: Multiple Fortune 500 companies are using TrueFoundry. Read about our case studies here.
  • Real-time usage analytics per user, model, and provider
  • Monitor latency, error rates, token usage, etc.
  • Cost Tracking and Optimization
  • Minimal latency overhead (less than 10ms)
  • Edge deployment capabilities
  • Rate limiting and abuse prevention
  • High availability architecture
🚀 Performance Benchmark: Learn how TrueFoundry AI Gateway delivers blazing-fast performance
Read our benchmark analysis →

Supported Integrations

We integrate with 1000+ LLMs through the following providers.
If you don’t see the provider you need, there is a high change it will just work as self hosted models or OpenAI provider. Please reach out to us at support@truefoundry.com and we will be happy to guide you.

Deployment Options

The Truefoundry AI Gateway can either be used as a SaaS offering or deployed on-premise.
  • SaaS Offering: You can directly use the gateway as a SaaS offering by signing up on our website, you can find the instructions here.
  • Enterprise Deployment for enterprise security and control. You can deploy the gateway in your cloud or on-premise. You can find the architecture and deployment instructions here.

Frequently Asked Questions

The latency overhead is minimal, typically less than 5ms. Our benchmarks show enterprise-grade performance that scales with your needs. Our SaaS offering is hosted in multiple regions across the world to ensure low latency and high availability. You can also deploy the gateway on-premise or on any cloud provider in your region which
is closer to your users.

AI Gateway on the edge, close to your applications for optimal performance

Yes, the AI Gateway supports on-premise deployments on any infrastructure or cloud provider, giving you complete control over your AI operations.
You can easily integrate any OpenAI-compatible self-hosted model. Check our self-hosted models guide for detailed instructions.
Yes, The AI Gateway can be used as a standalone solution. You can use the full MLOps platform if you’re using features like model deployment(traditional models and LLMs), model training, llm fine-tuning or training/data-processing workflows.

Feature Support Matrix

The following table shows which features are supported for each provider across different endpoints:
EndpointFeatureOpenAIAzure OpenAIAnthropicBedrockVertexCohereGeminiGroq
Chat CompletionStream
Chat CompletionNon Stream
Chat CompletionTool Support
Chat CompletionResponse Format: JSON Mode---
Chat CompletionResponse Format: Schema Mode
Chat CompletionPrompt Caching-
Chat CompletionReasoning Token
EmbeddingString---
EmbeddingList of String---
Image GenerateGenerate---
Image EditEdit---
Image VariationVariation-----
Audio TranscriptionTranscription---
Audio TranslationTranslation---
Text To SpeechSpeech---
RerankRerank------
BatchBatch
Fine-tuneFine Tune---
FilesFiles
ModerationModeration-----
Model ResponseModel Response-----

Legend

  • - Supported
  • - Provider support, We don’t
  • - - Provider does not support
I