Anthropic

TrueFoundry offers a secure and efficient gateway to seamlessly integrate various Large Language Models (LLMs) into your applications, including models available on Anthropic.

​

Adding Models

This section explains the steps to add Anthropic models and configure the required access controls.

• From the TrueFoundry dashboard, navigate to AI Gateway > Models and select Anthropic.
• Click Add Anthropic Account
• Complete the form with your Anthropic account details, including the authentication information API Key.
• Select the model from the list.
• (Optional) If the model you are looking for is not present in the options, you can add it using + Add Model at the end of list by filling the form

• Select the Anthropic Model option and provide the model ID and other required details to add one or more model integrations.

​

Inference

After adding the models, you can perform inference using an Anthropic-compatible API via the TrueFoundry AI Gateway. For instance, you can directly utilize the Anthropic library as well.:

from openai import OpenAI

client = Anthropic(api_key="Enter your API Key here", base_url="https://llm-gateway.truefoundry.com/api/inference/openai")
stream = client.chat.completions.create(
    messages = [
            {"role": "system", "content": "You are an AI bot."},
            {"role": "user", "content": "Enter your prompt here"},
    ],
    model= "<PROVIDER_ACCOUNT_NAME>/<MODEL_NAME>",
)

​

Supported Models

A list of models supported by AWS Anthropic, along with their corresponding model IDs, can be found here: View Full List

The TrueFoundry AI Gateway supports all text and image models in Anthropic. We are also working on adding support for additional modalities, including speech, in the near future.

​

Extra Parameters

Internally, the TrueFoundry AI Gateway utilizes the Anthropic API for chat completion.

To pass additional input fields or parameters, such as top_k, frequency_penalty, and others specific to a model, include them using this key:

"additionalModelRequestFields": {
    "frequency_penalty": 0.5
}