Deploy LLM Gateway Only

Architecture

The architecture of Truefoundry Gateway comprises the truefoundry helm chart, which comprises of the following components:

1. Frontend
2. Backend Microservice
3. Postgres DB to store all metadata - this is created on Kubernetes in dev mode. However, for production setups, we recommend using a managed database like AWS RDS or GCP Cloud SQL.
4. Queue to handle request logging and configuration propagation.
5. Clickhouse DB to store all request logs.

The helm chart can be completely deployed within your own VPC. This will talk to the Truefoundry authentication/licensing server in Truefoundry VPC to validate the license key and users. The only information that is communicated to the central licensing server is the user's email address who logs into the platform and analytics on the number of requests made through the gateway.

Prerequisites for Installation

1. Egress access to auth.truefoundry.com and analytics.truefoundry.com
2. Domain to map the ingress of the frontend and gateway
3. Tenant, Licence key, and image pull secret from TrueFoundry team

Installation Instructions

1. Create a values file as given below and replace relevant values:
   yaml

   global:
     # Ask TrueFoundry team to provide these
     tenantName: <TENANT_NAME>
     tfyApiKey: <LICENCE_KEY>
     truefoundryImagePullConfigJSON: <TRUEFOUNDRY_IMAGE_PULL_SECRET>
     
     # Domain to map the platform to
     controlPlaneURL: https://example.com
     resourceTier: small
   
   truefoundryFrontendApp:
     # Choose the service mesh: we support both istio and k8s ingress, enable as per your setting
     # Replace `example.com` with your Domain
     istio:
       virtualservice:
         hosts:
           - example.com
         enabled: false
         gateways:
           - istio-system/tfy-wildcard
     ingress:
       hosts:
         - example.com
       enabled: false
       annotations: {}
       ingressClassName: nginx
   tags:
     llmGateway: true
     llmGatewayRequestLogging: true
   # Enable devMode
   devMode:
     enabled: true
   # Disable few depenencies for only LLM Gateway setup
   tfyBuild:
     enabled: false
   tfyController:
     enabled: false
   mlfoundryServer:
     enabled: false
   tfy-buildkitd-service:
     enabled: false

2. Add helm repo
   bash

   helm repo add truefoundry https://truefoundry.github.io/infra-charts

3. Install the helm chart with your values file
   bash

   helm upgrade --install truefoundry truefoundry/truefoundry -n truefoundry --create-namespace -f truefoundry-values.yaml