Overview

Adding Models

To add Google Vertex AI models to TrueFoundry LLM Gateway:

1. From TrueFoundry dashboard, go to Integrations > Add Provider Integration and choose Google
2. Add Authentication details by providing Service Account JSON. More details on permission required below.
3. Click on Vertex Model and fill in model ID, region and other fields to add model integrations. Read below for more information on how to obtain the model ID from Vertex.

Authentication

To authenticate Google Vertex model you need to provide:

1. Project ID - Project ID can be found from the top right corner in your console.
   1. 
2. Service Account JSON - You can generate a service account JSON using the steps mentioned here.
3. The corresponding service account needs to have an IAM role with the following permission:

   1. [
        "aiplatform.endpoints.predict"
      ]

Supported Models

Currently TrueFoundry LLM Gateway supports text and embedding models from Vertex AI.

Some of these models support function calling and sending files like images or documents attached to their messages and TrueFoundry LLM Gateway also supports the same. Support for Vertex AI models like image generation, audio are coming soon on TrueFoundry LLM Gateway.

Model ID

While adding models to LLM Gateway, you need to provide the Vertex Model ID. This can be obtained from the models page on Vertex.

While adding Google models like Gemini or Gemma, you can simply provide the model ID in the form. For example, you'd fill gemini-1.5-flash for Gemini 1.5 Flash model.

While adding models from other Vertex partners like Anthropic or Mistral, you need to provide the model ID in the <partner>/<vertex-model-id. For example, anthropic/claude-3-5-sonnet-v2@20241022.