LLM Gateway

Follow the steps to enable LLM Gateway in your control plane:

1. For enabling request logging and metrics in the LLM gateway, few infra components are required like nats and clickhouse. Follow these steps to install: (Note: password should not contain @ or :)
   1. Create k8s secret

      apiVersion: v1
      kind: Secret
      metadata:
        name: tfy-llm-gateway-infra-auth
        namespace: truefoundry
      stringData:
        CLICKHOUSE_USER_PASSWORD: xxx
        NATS_ADMIN_PASSWORD: xxx
        NATS_CLICKHOUSE_REQUEST_LOGS_READER_PASSWORD: xxx
        NATS_LLM_GATEWAY_REQUEST_LOGGER_PASSWORD: xxx
      type: Opaque

   2. Install tfy-llm-gateway-infra helm chart via truefoundry

      name: tfy-llm-gateway-infra
      type: helm
      source:
       type: helm-repo
       chart: tfy-llm-gateway-infra
       version: 0.2.2
       repo_url: https://truefoundry.github.io/infra-charts/

2. Install tfy-llm-gateway helm chart

   name: tfy-llm-gateway
   type: helm
   source:
     type: helm-repo
     chart: tfy-llm-gateway
     version: 0.18.2
     repo_url: https://truefoundry.github.io/infra-charts/
   values:
     global:
       existingTruefoundryImagePullSecretName: truefoundry-image-pull-secret
       controlPlaneURL: <>
       llmGatewayInfra:
         enabled: true
         natsAdminPassword: "${k8s-secret/tfy-llm-gateway-infra-auth/NATS_ADMIN_PASSWORD}" # to refer k8s secret in step 1

3. Update the control plane (truefoundry helm chart) configuration - please add the following section under global in values

   values:
     ...
     global:
       ...
       llmGatewayInfra:
         enabled: true
         natsAdminPassword: "${k8s-secret/tfy-llm-gateway-infra-auth/NATS_ADMIN_PASSWORD}" # to refer k8s secret in step 1
         clickhousePassword: "${k8s-secret/tfy-llm-gateway-infra-auth/CLICKHOUSE_USER_PASSWORD}" # to refer k8s secret in step 1