Key Components
To install the complete control plane on your own infrastructure, you need to install the following components:- Truefoundry Control Plane + Gateway (Shipped as a single helm chart called
truefoundry
) - PostgreSQL Database (Managed or Self-Hosted with PostgreSQL >= 13)
- Blob Storage (S3, GCS, Azure Container or any other S3 compatible storage)
Compute Requirements
Truefoundry ships as a helm chart (https://github.com/truefoundry/infra-charts/tree/main/charts/truefoundry) that has configurable options to either deploy both Deployment and AI Gateway feature or just choose the one of them according to your needs. The compute requirements change based on the set of features and the scale of the number of users and requests. Here are a few scenarios that you can choose from based on your needs.The small tier is recommended for development purposes. Here all the components are deployed on Kubernetes and in non HA mode (single replica). This is suitable if you are just testing out the different features of Truefoundry.
This setup brings up 1 replica of the services and is not highly-available. It
can enable you to test the features but we do not recommend this for
production mode.
Component | CPU | Memory | Storage | Min Nodes | Remarks |
---|---|---|---|---|---|
Helm-Chart (AI Gateway Control Plane components) | 2 vCPU | 8GB | 60GB Persistent Volumes (Block Storage) On Kubernetes | 2 Pods should be spread over min 2 nodes | Cost: ~ $120 pm |
Helm-Chart (AI Gateway component only) | 1 vCPU | 512Mi | - | 1 Pods should be spread over min 1 node | Cost: ~ $10 pm |
Postgres (Deployed on Kubernetes) | 0.5 vCPU | 0.5GB | 5GB Persistent Volumes (Block Storage) On Kubernetes | PostgreSQL version >= 13 IOPS: Default (suitable for dev/testing) For PostgreSQL 17+: Disable SSL, for AWS: by setting force_ssl parameter to 0 in the parameter group, for Azure: by setting require_secure_transport parameter to false in the parameter group | |
Blob Storage (S3 Compatible) | 20GB |
Prerequisites for Installation
- Kubernetes Cluster: K8s cluster 1.27+.
- Support for dynamic provisioning of storage for PVC (for e.g AWS EBS, Azure Disk etc.) and support for ingress controller (for e.g. Nginx Ingress Controller) or istio service mesh for exposing the control plane dashboard and AI Gateway at an endpoint.
- Domain to map the ingress of the Control Plane dashboard and AI Gateway along with certificate for the domain.
This Domain will be referred as Control Plane URL in our documentation.
- Egress access to TrueFoundry Central Auth Server: https://auth.truefoundry.com & https://login.truefoundry.com
- Tenant Name, Licence key, and image pull secret from TrueFoundry team. If you have not registered yet, please visit TrueFoundry to register.
- PostgreSQL database. We usually recommend managed PostgreSQL database (For e.g. AWS RDS, or Google Cloud SQL, or Azure Database for PostgreSQL) for production environments. For instance requirements, refer to the Compute Requirements section.
In case, you do not have a managed database just for testing purposes, set
devMode
to true in the values file to spin up a local PostgreSQL database. - Blob Storage to store the AI Gateway request logs (either S3, GCS, Azure Blob Storage, or any other S3 compatible storage). You can find the instructions in the guide below.
Installation Instructions
1
Setup Control Plane Platform IAM Role
Creating AWS IAM Role for Control Plane
Creating AWS IAM Role for Control Plane
Control Plane IAM Role needs to have permission to assume any other IAM role in or cross account to provide access to different cloud services like blob, secret, models, etc.
- Create a new IAM role for Control Plane with a suitable name like
tfy-control-plane-platform-deps
- Following is the IAM policy that needs to be attached to the Control Plane IAM Role:
Here ”*” is used to allow the Control Plane IAM Role to assume any other IAM role in or cross account. In place of ”*” you can also give specific ARNs of other IAM roles
- Add the following trust policy to the Control Plane IAM Role:
In place of
<ACCOUNT_ID>
, <AWS_REGION>
, and <OIDC_ID>
you can also give the values from your EKS cluster.
You can find the OIDC_ID from the EKS cluster. Also, here we are assuming that the service account is truefoundry
and the namespace is truefoundry
, you can change it as per your needs.Allow your IAM role to be assumed by Control Plane IAM Role
- Add the following trust policy to your IAM Role to allow it to be assumed by the Control Plane IAM Role:
2
Create S3 Bucket
Create a S3 Bucket with following config:Create a IAM Policy to allow access to the S3 Bucket with following config:Attach the IAM Policy to the Control Plane Platform IAM Role
- Make sure the bucket has lifecycle configuration to abort multipart upload set for 7 days.
- Make sure CORS is applied on the bucket with the below configuration:
3
Create Postgres RDS Database
Create a Postgres RDS instance of size
db.t3.medium
with storage size of 30GB.Important Configuration Notes: - For PostgreSQL 17: Disable SSL by
setting
force_ssl
parameter to 0
in the parameter group - Security
Group: Ensure your RDS security group has inbound rules allowing traffic
from EKS node security groupsIn case you want to setup PostgreSQL on Kubernetes and not use RDS for testing purposes, skip this step and set
devMode
to true in the values file below4
Create Kubernetes Secrets
We will create two secrets in this step:
- Store the License Key and DB Credentials
- Store the Image Pull Secret
Create Kubernetes Secret for License Key and DB Credentials
Create Kubernetes Secret for License Key and DB Credentials
We need to create a Kubernetes secret containing the licence key and db credentials.Apply the secret to the Kubernetes cluster (Assuming you are installing the control plane in the
If you are using PostgreSQL on Kubernetes in the dev mode, the values will be as follows:DB_HOST: <HELM_RELEASE_NAME>-postgresql.<NAMESPACE>.svc.cluster.local // eg. truefoundry-postgresql.truefoundry.svc.cluster.localDB_NAME: truefoundryDB_USERNAME: postgres # In order to use custom username, please update the same at
postgresql.auth.username
DB_PASSWORD: randompassword # You can change this to any value here.truefoundry-creds.yaml
truefoundry
namespace)Create Kubernetes Secret for Image Pull Secret
Create Kubernetes Secret for Image Pull Secret
We need to create a Image Pull Secret to enable pulling the truefoundry images from the private registry.Apply the secret to the Kubernetes cluster (Assuming you are installing the control plane in the
truefoundry-image-pull-secret.yaml
truefoundry
namespace)5
Create HelmChart Values file
Create a values file as given below and replace the following values:
- Control Plane URL: URL that you will map to the control plane dashboard (e.g.,
https://truefoundry.example.com
) - Tenant Name: Tenant name provided by TrueFoundry team
- AWS S3 Bucket Name: Name of the S3 bucket you created in the previous step (e.g.,
my-truefoundry-bucket
) - AWS Region: Region of the S3 bucket you created in the previous step (e.g.,
us-west-2
) - Control Plane IAM Role ARN: ARN of the IAM role you created in the previous step (e.g.,
arn:aws:iam::123456789012:role/tfy-control-plane-platform-deps
)
truefoundry-values.yaml
6
Install Helm chart
FAQ
Can I use the images from my own private registry?
Can I use the images from my own private registry?
If you want to use private image registry for the LLM Gateway, you can do so by following the instructions below.1. Push all the required imaged to your private image registry.
There are two ways to do this:2. Update the
You can use your Artifactory as a mirror to pull images from our registry. The artifactory details for Truefoundry are as follows:Registry URL:
https://tfy.jfrog.io/
To get the username and password, you can base64 decode the Image Pull Secret that you receive from Truefoundry team.truefoundry-values.yaml
file to use your custom images.Can I use my Artifactory as a mirror to pull images?
Can I use my Artifactory as a mirror to pull images?
Yes, you can use your Artifactory as a mirror to pull images from our registry. The artifactory details for Truefoundry are as follows:Registry URL:
https://tfy.jfrog.io/
To get the username and password, you can base64 decode the Image Pull Secret that you receive from Truefoundry team.Do we need any NFS volumes in Kubernetes for the AI Gateway or Control Plane?
Do we need any NFS volumes in Kubernetes for the AI Gateway or Control Plane?
No, we only need block storage for installing and running Truefoundry. This should be supported via the CSI driver and only ReadWriteOnce access is required.
What is the structure of access logs
What is the structure of access logs
We log access information in standard output with the following format:
Examples:
- logfmt
- json
Log format
Standard log format structure:Log operator | Details |
---|---|
START_TIME | ISO timestamp for request start. eg. 2025-08-12 13:34:50 |
LEVEL | info|warn|error |
IP_ADDRESS | IP address of the caller. eg. ::ffff:10.99.55.142 |
TENANT_NAME | Name of the tenant. eg. truefoundry |
SUBJECT_TYPE | user|virtualaccount |
SUBJECT_SLUG | Email or virtual account name. eg. tfy-user@truefoundry.com|demo-virtualaccount |
MODEL_ID | Model ID. eg. openai-default/gpt-5 |
METHOD | GET|POST|PUT |
PATH | Path of the request. eg. /api/inference/openai/chat/completions |
STATUS_CODE | 200|400|401|403|429|500 |
DURATION | Duration of the request. eg. 12 |
TRACE_ID | Trace ID of the request |
How to use SSO directly without using TrueFoundry Auth Server?
How to use SSO directly without using TrueFoundry Auth Server?
By default, the control plane uses TrueFoundry managed Central Auth server to authenticate users. However, you can also deploy the control plane without using the TrueFoundry managed Central Auth server.
We currently support any OIDC compliant Identity Provider to be used as external identity provider.You need to add your OIDC application details under
servicefoundryServer.env
in the values.yaml
file of truefoundry
helm installation.Here we are assuming the identity provider is OIDC compliant and satisfies following:
- Openid configuration is available at
<ISSUER_URL>/.well-known/openid-configuration
. - Scopes configured should include
openid
,email
,profile
andoffline_access
. - Allowed Redirect URI should be set to
<CONTROL_PLANE_URL>/auth/callback
. - OIDC issuer servers should be accessible from user’s browser, TrueFoundry control plane servers and all multi zone AI Gateway servers.