Deployment Options
| Deployment Scenario | Modules | Pricing Tier | Remarks | Cost of Hosting |
|---|---|---|---|---|
| 1. AI Gateway SAAS only | Gateway | Starter | Use managed AI Gateway for LLM requests, no compute plane or control plane hosting needed. | 0 |
| 2. Saas AI Gateway with data storage on your own infrastructure | Gateway | Starter | Use managed AI Gateway for LLM requests, no compute plane or control plane hosting needed. | Storage cost in S3 |
| 3. Gateway Plane only | Gateway | Enterprise | Host the Gateway plane for LLM requests and use control-plane hosted by Truefoundry. | ~$600/month |
| 4. Control Plane + Gateway Plane | AI Gateway | Enterprise | Use only AI Gateway, full control, suitable if you don’t need compute plane/app deployment. | ~ $800-1000/month. |
Option 1: AI Gateway SAAS only
This is a fully managed solution on Truefoundry’s secure cloud infrastructure with enterprise-grade features. You don’t need to deploy any infrastructure on your end. This includes usage of only the AI Gateway modle and not AI Deployment.
Truefoundry Managed SAAS
- Globally distributed gateway to minimize latency: Truefoundry gateway is deployed in multiple regions of the world across multiple zones and multiple cloud providers to provide low latency and high availability. Learn more about our globally distributed infrastructure.
- Zero Overhead of maintenance: There is no overhead of maintaining infrastructure and you can get access to the latest features and improvements.
- Data is encrypted at rest and in transit.
- Truefoundry Infrastructure is SOC2, ISO27001, GDPR, and HIPAA compliant
Option 2: Saas AI Gateway with data storage on your own infrastructure
In this case, the gateway and control-plane are managed by Truefoundry on its own infrastructure, but the LLM data is stored on your own infrastructure.
Truefoundry Managed Gateway with data storage on your own infrastructure
- No Infrastructure Management: In this case, the gateway is globally distributed, available and fully managed by Truefoundry. Learn more about our globally distributed infrastructure. You just need to provide the blob storage(AWS S3 Bucket, or GCS Bucket or Azure Blob Storage or any other S3 compatible storage) and the control-plane will use it to store the LLM data.
- You retain data ownership: You retain full ownership of the request-response data since the its stored on a bucket on your end. The data are stored in parquet format - so you can use them for analytics, debugging and evaluation via Spark, DuckDB, Athena or any tool of your choice.
The actual request response data in this case flows through Truefoundry control-plane and then gets stored in a blob storage on your infrastructure. The control-plane might cache some of the data for faster queries, but doesn’t do any long term retention of the data.When you are browsing the request logs in the Truefoundry dashboard, the data will be fetched from your blob storage - so you might incur egress charges from your cloud provider.
Option3: Gateway Plane on your own infrastructure
This is a deployment of the gateway plane on your own infrastructure. The control-plane is hosted by Truefoundry. The gateway exports the request-response data to the ingestion server which then stores the data in your own blob storage. The control-plane stores the metrics and has access to the bucket containing the request-response data.
Gateway Plane and Data Storage on your own infrastructure
- LLM Traffic stays within your own premises: All LLM traffic stays within your own infrastructure and Truefoundry doesn’t come into the live path of a request to LLM.
- You retain full control over your data: You retain full ownership of the request-response data since the its stored on a bucket on your end. The data are stored in parquet format - so you can use them for analytics, debugging and evaluation via Spark, DuckDB, Athena or any tool of your choice.
- Management of Gateway and Ingestion Service: The gateway and ingestion service availability needs to be managed on your end.
- Truefoundry control plane has access to the bucket containing the data: This access helps you browse the request logs on the truefoundry dashboard. The control-plane also need to access the logs to be able to compact it and create indexes on it to make it faster to query.
You will not be be able to use this feature if you don’t give access to Truefoundry control plane access to your bucket.
When you are browsing the request logs in the Truefoundry dashboard, the data will be fetched from your blob storage - so you might incur egress charges from your cloud provider. The data might be cached temporarily in the control-plane for faster queries. There are also egress charges when the control-plane compacts the logs and creates indexes on it.
Option4: Control Plane + Gateway Plane
Host both the gateway plane and the control plane on your own infrastructure. In this case, everything except the authentication server and analytics server, everything is hosted on your own infrastructure.
Control Plane, Gateway Plane and Data Storage on your own infrastructure
The only data sent to authentication/licensing server are the emails of the employees using the platform and the count of the requests flowing through the gateway. To understand how SSO works with our central authentication server, refer to this page. This helps us keep track of the licenses and billing.