Skip to main content

​
Adding Models

This section explains the steps to add AWS Bedrock models and configure the required access controls.
1

Navigate to AWS Bedrock Models in AI Gateway

From the TrueFoundry dashboard, navigate to AI Gateway > Models and select AWS Bedrock.
Navigating to AWS Bedrock Provider Account in AI Gateway

Navigate to AWS Bedrock Models

2

Add AWS Bedrock Account Name and Collaborators

Give a unique name for the bedrock account which will be used to refer later in the models. The models in the account will be referred to as @providername/@modelname. Add collaborators to your account. You can decide which users/teams have access to the models in the account (User Role) and who can add/edit/remove models in this account (Manager Role). You can read more about access control here.
AWS Bedrock account configuration form with fields for API key and collaborators

AWS Bedrock Model Account Form

3

Add Region and Authentication

Select the default AWS region for the models in this account. **The account-level region serves as the default for all models unless explicitly overridden at the model level. **Provide the authentication details on how the gateway can access the Bedrock models. Truefoundry supports both AWS Access Key/Secret Key and Assume Role based authentication. You can read below on how to generate the access/secret keys or roles.
Using AWS Access Key and Secret
  1. Create an IAM user (or choose an existing IAM user) following these steps.
  2. Add required permission for this user. The following policy grants permission to invoke all model in your available regions (To check the list of available regions for different models, refer to AWS Bedrock).
  3. {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Sid": "InvokeAllModels",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": [
		"arn:aws:bedrock:*:<aws-account-id>:foundation-model/*",
        "arn:aws:bedrock:*:<aws-account-id>:inference-profile/*",
        "arn:aws:bedrock:*:<aws-account-id>:application-inference-profile/*"
	  ]
    }
  ]
}
  4. Create an access key for this user as per this doc.
  5. Use this access key and secret while adding the provider account to authenticate requests to the Bedrock model.
Using Assumed Role
  1. You can also directly specify a role that can be assumed by the service account attached to the pods running AI Gateway.
  2. Read more about how assumed roles work here.
4

Add Models

Select the models from the list that you want to add. You can use Select All to select all the models.
If the model you are looking for is not present in the options, you can add it using + Add Model at the end of list.
TrueFoundry AI Gateway supports all text and image models in Bedrock.The complete list of models supported by Bedrock can be found here.

​
Inference

After adding the models, you can perform inference using an OpenAI-compatible API via the Playground or integrate with your own application.
Code Snippet and Try in Playgroud Buttons for each model

Infer Model in Playground or Get Code Snippet to integrate in your application

​
FAQ:

In case you have custom pricing for your models, you can override the default cost by clicking on Edit Model button and then choosing the Private Cost Metric option.
Edit model button and interface for AWS Bedrock model

Edit Model

Custom cost metric configuration form with input fields for pricing

Set custom cost metric

Yes, you can add models from different regions. You can provide a top level default region for the account and also override it at the model level.Region selection dropdown for AWS Bedrock model configuration
AWS Bedrock Cross-Region Inference is a feature that allows you to use models from different regions and can automatically route the request to the best region based on factors like Current load and capacity in each region, Network latency and performance, Regional availability and health. You can read more about it here. You can also read more on which models support cross-region inference here.AWS Bedrock cross-region inference configuration interface

​
To use cross region inference models in Truefoundry, you can use the Inference Profile ID instead of the model ID.

When using system-defined inference profiles for cross-Region routing, please ensure you grant model access permission in all destination regions to the role or access/secret key provided in Truefoundry while doing the integration. Failure to do so will result in failed inference requests if Bedrock decides to route the request to a different region (apart from the source region) because of lack of access in the other region.
⌘I