Follow the steps to enable AI Gateway in your control plane:

  1. For enabling request logging and metrics in the AI gateway, few infra components are required like nats and clickhouse. Follow these steps to install: (Note: password should not contain @ or :)

    1. Create k8s secret 
      apiVersion: v1
kind: Secret
metadata:
  name: tfy-llm-gateway-infra-auth
  namespace: truefoundry
stringData:
  CLICKHOUSE_USER_PASSWORD: xxx
  NATS_ADMIN_PASSWORD: xxx
  NATS_CLICKHOUSE_REQUEST_LOGS_READER_PASSWORD: xxx
  NATS_LLM_GATEWAY_REQUEST_LOGGER_PASSWORD: xxx
type: Opaque
    2. Install tfy-llm-gateway-infra helm chart via truefoundry 
      name: tfy-llm-gateway-infra
type: helm
source:
 type: helm-repo
 chart: tfy-llm-gateway-infra
 version: 0.2.2
 repo_url: https://truefoundry.github.io/infra-charts/

  2. Install tfy-llm-gateway helm chart

    name: tfy-llm-gateway
type: helm
source:
  type: helm-repo
  chart: tfy-llm-gateway
  version: 0.18.2
  repo_url: https://truefoundry.github.io/infra-charts/
values:
  global:
    existingTruefoundryImagePullSecretName: truefoundry-image-pull-secret
    controlPlaneURL: <>
    llmGatewayInfra:
      enabled: true
      natsAdminPassword: "${k8s-secret/tfy-llm-gateway-infra-auth/NATS_ADMIN_PASSWORD}" # to refer k8s secret in step 1

  3. Update the control plane (truefoundry helm chart) configuration - please add the following section under global in values

    values:
  ...
  global:
    ...
    llmGatewayInfra:
      enabled: true
      natsAdminPassword: "${k8s-secret/tfy-llm-gateway-infra-auth/NATS_ADMIN_PASSWORD}" # to refer k8s secret in step 1
      clickhousePassword: "${k8s-secret/tfy-llm-gateway-infra-auth/CLICKHOUSE_USER_PASSWORD}" # to refer k8s secret in step 1