AWS Bedrock

Overview

Adding Models

To add Bedrock model to Gateway:

  • From TrueFoundry dashboard, go to Integrations > Add Provider Integration and choose AWS
  • Fill out the form with AWS account details, authentication data (Access key + Secret) or Assume Role ID.

  • Click on Bedrock Model and fill out model ID and other details to add one or more model integrations.

Inference

Once models are added, you can run inference on these models using an OpenAI-compatible API through the TrueFoundry LLM Gateway. For example, you can directly use the OpenAI library too:

from openai import OpenAI client = OpenAI(api_key="Enter your API Key here", base_url="https://llm-gateway.truefoundry.com/api/inference/openai") stream = client.chat.completions.create( messages = [ {"role": "system", "content": "You are an AI bot."}, {"role": "user", "content": "Enter your prompt here"}, ], model= "bedrock-provider/llama-70b", )

Supported Models

List of models supported by Bedrock and their corresponding model IDs can be found here: https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html#model-ids-arns

TrueFoundry LLM Gateway supports all text and image models in Bedrock. We are adding support for other modalities including speech shortly.

Extra Parameters

Internally, TrueFoundry LLM Gateway uses Bedrock Converse API for chat completion.

If you need to pass additional input fields or parameters like top_k frequency_penalty etc. that are specific to a model, you can pass it with this key:

"additionalModelRequestFields": { "frequency_penalty": 0.5 }

Cross-Region Inference

To manage traffic during on-demand inferencing by utilising compute across regions, you can use AWS Bedrock Cross-Region Inference. While setting model ID in TrueFoundry, use the Inference Profile ID instead of the model ID.

You can more information about cross-region inferencing here.

Use Inference Profile ID as model ID while adding model to TFY LLM Gateway

Authentication Methods

Using AWS Access Key and Secret

  1. Create an IAM user (or choose an existing IAM user) following these steps.
  2. Add required permission for this user. The following policy grants permission to invoke all model
    1. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "InvokeAllModels", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/*" ] } ] }
  3. Create an access key for this user as per this doc.
  4. Use this access key and secret while adding the provider account to authenticate requests to the Bedrock model.

Using Assumed Role

  1. You can also directly specify a role that can be assumed by the service account attached to the pods running LLM Gateway.
  2. Read more about how assumed roles work here.

Did this page help you?