Deploy AI Gateway

Key Components

To install the complete control plane on your own infrastructure, you need to install the following components:

Truefoundry Control Plane + Gateway (Shipped as a single helm chart called truefoundry)
Postgres Database (Managed or Self-Hosted with Postgres >= 13)
Blob Storage (S3, GCS, Azure Container or any other S3 compatible storage)

Compute Requirements

Truefoundry ships as a helm chart (https://github.com/truefoundry/infra-charts/tree/main/charts/truefoundry) that has configurable options to either deploy both Deployment and AI Gateway feature or just choose the one of them according to your needs. The compute requirements change based on the set of features and the scale of the number of users and requests. Here are a few scenarios that you can choose from based on your needs.

The small tier is recommended for development purposes. Here all the components are deployed on Kubernetes and in non HA mode (single replica). This is suitable if you are just testing out the different features of Truefoundry.

This setup brings up 1 replica of the services and is not highly-available. It can enable you to test the features but we do not recommend this for production mode.

Component	CPU	Memory	Storage	Min Nodes	Remarks
Helm-Chart (AI Gateway Control Plane components)	2 vCPU	8GB	60GB _{Persistent Volumes (Block Storage) On Kubernetes}	2 _{Pods should be spread over min 2 nodes}	_{Cost: ~ $120 pm}
Helm-Chart (AI Gateway component only)	1 vCPU	512Mi	-	1 _{Pods should be spread over min 1 node}	_{Cost: ~ $10 pm}
Postgres (Deployed on Kubernetes)	0.5 vCPU	0.5GB	5GB _{Persistent Volumes (Block Storage) On Kubernetes}		PostgreSQL version >= 13
Blob Storage (S3 Compatible)			20GB

Prerequisites for Installation

Egress access to TrueFoundry Central Auth Server: https://auth.truefoundry.com
Domain to map the ingress of the Control Plane dashboard and AI Gateway along with certificate for the domain.
This Domain will be referred as Control Plane URL in our documentation.
Tenant Name, Licence key, and image pull secret from TrueFoundry team. If you have not registered yet, please visit TrueFoundry to register.
In case of AWS or GCP, you will also need to create a IAM role to act as Workload Identity for the Control Plane.

Setup Control Plane Platform IAM Role

Creating AWS IAM Role for Control Plane

Control Plane IAM Role needs to have permission to assume any other IAM role in or cross account to provide access to different cloud services like blob, secret, models, etc.

Create a new IAM role for Control Plane with a suitable name like tfy-control-plane-platform-deps
Following is the IAM policy that needs to be attached to the Control Plane IAM Role:

{
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}

Here ”*” is used to allow the Control Plane IAM Role to assume any other IAM role in or cross account. In place of ”*” you can also give specific ARNs of other IAM roles

Add the following trust policy to the Control Plane IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
              "StringEquals": {
                  "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": [
                      "system:serviceaccount:truefoundry:servicefoundry-server",
                      "system:serviceaccount:truefoundry:truefoundry",
                  ]
              }
          }
      }
  ]
}

In place of <ACCOUNT_ID>, <AWS_REGION>, and <OIDC_ID> you can also give the values from your EKS cluster. You can find the OIDC_ID from the EKS cluster. Also, here we are assuming that the service account is truefoundry and the namespace is truefoundry, you can change it as per your needs.

Allow your IAM role to be assumed by Control Plane IAM Role

Add the following trust policy to your IAM Role to allow it to be assumed by the Control Plane IAM Role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<CONTROL_PLANE_IAM_ROLE_ARN>"
            },
            "Action": "sts:AssumeRole"
        }
      ]
}

Blob Storage to store the AI Gateway request logs.

Setup Blob Storage

To setup AWS S3 for control plane, follow the steps below:

Create a S3 Bucket with following config:
- Make sure the bucket has lifecycle configuration to abort multipart upload set for 7 days.
- Make sure CORS is applied on the bucket with the below configuration:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

Create a IAM Policy to allow access to the S3 Bucket with following config:

{
  "Sid": "S3",
  "Effect": "Allow",
  "Action": ["s3:*"],
  "Resource": [
    "arn:aws:s3:::<YOUR_S3_BUCKET_NAME>",
    "arn:aws:s3:::<YOUR_S3_BUCKET_NAME>/*"
  ]
}

Attach the IAM Policy to the Control Plane Platform IAM Role

(Optional) Managed Postgres database. For instance requirements, refer to the Compute Requirements section.
In case, you do not have a managed database, set devMode to true in the values file to spin up a local postgres database.

Installation Instructions

Create a values file as given below and replace relevant values:

global:
  # Choose the resource tier as per your needs
  resourceTier: medium # or small or large

  # Domain to map the platform to
  controlPlaneURL: https://example.com

  # Ask TrueFoundry team to provide these
  tenantName: <TENANT_NAME>

  # Pass the secret name containing the licence key and db credentials. Follow the [docs](https://github.com/truefoundry/infra-charts/blob/main/charts/truefoundry/README.md#using-k8s-secret-for-required-fields) to create the secret.
  # Must contain the following keys:
  # TFY_API_KEY - Licence key
  # DB_HOST - Postgres host
  # DB_NAME - Postgres database name
  # DB_USERNAME - Postgres username
  # DB_PASSWORD - Postgres password
  existingTruefoundryCredsSecret: "truefoundry-creds"
  
  # Pass the secret name containing the image pull secret. Follow the [docs](https://github.com/truefoundry/infra-charts/blob/main/charts/truefoundry/README.md#using-k8s-secret-for-required-fields) to create the secret.
  existingTruefoundryImagePullSecretName: "truefoundry-image-pull-secret"

  config:
    defaultCloudProvider: "aws"
    storageConfiguration:
      awsS3BucketName: '<AWS_S3_BUCKET_NAME>'
      awsRegion: '<AWS_REGION>'

  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: <CONTROL_PLANE_IAM_ROLE_ARN>

# In case, you want to spin up postgres on kubernetes, enable this
devMode:
  enabled: false

truefoundryFrontendApp:
  # Choose the service mesh: we support both istio and k8s ingress, enable as per your setting
  # Replace `example.com` with your Domain
  istio:
    virtualservice:
      hosts:
        - example.com
      enabled: false
      gateways:
        - istio-system/tfy-wildcard
  ingress:
    hosts:
      - example.com
    enabled: false
    annotations: {}
    # For haproxy openshift, please add the following annotation to the ingress
    # haproxy.router.openshift.io/rewrite-target: /
    ingressClassName: nginx
tags:
  llmGateway: true
  llmGatewayRequestLogging: true

# For enabling read only root filesystem in clickhouse, please add the following:
# tfy-clickhouse:
#   clickhouse:
#     podSecurityContext:
#       fsGroup: <ALLOWED_FS_GROUP_ID>
#       runAsUser: <ALLOWED_USER_ID>
#       runAsGroup: <ALLOWED_GROUP_ID>
#       capabilities:
#         add:
#           - CAP_NICE
#           - CAP_IPC_LOCK
#     containerSecurityContext:
#       runAsUser: <ALLOWED_USER_ID>
#       runAsGroup: <ALLOWED_GROUP_ID>
#       capabilities:
#         drop:
#           - ALL
#       runAsNonRoot: true
#       seccompProfile:
#         type: RuntimeDefault
#       allowPrivilegeEscalation: false

# Disable few depenencies for only LLM Gateway setup
tfyBuild:
  enabled: false
sfyManifestService:
  enabled: false
tfyController:
  enabled: false
mlfoundryServer:
  enabled: false
tfy-buildkitd-service:
  enabled: false

Install the helm chart with your values file

helm upgrade --install truefoundry oci://tfy.jfrog.io/tfy-helm/truefoundry -n truefoundry --create-namespace -f truefoundry-values.yaml

Get Started

Developer Guide

Configure Gateway

MCP Registry and Gateway

Observability

Integrations

Deployment

API Reference

Chat

Agent Responses

Embeddings

Rerank

Responses

Audio

Batch

Files

Moderations

Key Components

Compute Requirements

Prerequisites for Installation

Creating AWS IAM Role for Control Plane

Allow your IAM role to be assumed by Control Plane IAM Role

Installation Instructions

Get Started

Developer Guide

Configure Gateway

MCP Registry and Gateway

Observability

Integrations

Deployment

API Reference

Chat

Agent Responses

Embeddings

Rerank

Responses

Audio

Batch

Files

Moderations

​Key Components

​Compute Requirements

​Prerequisites for Installation

​Installation Instructions

Key Components

Compute Requirements

Prerequisites for Installation

Installation Instructions