OpenAI, Anthropic, DeepSeek and More

TrueFoundry offers a secure and efficient gateway to seamlessly integrate various Large Language Models (LLMs) into your applications, including models hosted on OpenAI, DeepSeek, Anthropic, AWS Bedrock, Google Vertex, Azure OpenAI and many more. View Full List

Providers and Models

Before we start integrating LLMs, let's quickly understand the concept of a Provider and a Model:

Provider - A provider is an organization or company that supplies access to various AI language models. These providers offer multiple models, each varying in performance, features, and pricing. Prominent examples of such providers include OpenAI and Anthropic, which deliver a range of models designed to meet different needs.

Model - A model refers to a specific implementation of an AI language model made available by a provider. Each model is tailored with unique features and trained on distinct datasets, which influences its performance and suitability for various tasks. For example, OpenAI's GPT-3 and GPT-4 are different models, each offering unique capabilities to cater to diverse applications.

Steps to Add OpenAI Models to Gateway

This section outlines the process of adding OpenAI models and configuring the necessary access controls.

Access Control

In the example above, all users have access to every model. However, access control can be configured at the individual model level.

Adding access control through the UI

When adding a model, you have the option to configure which users and teams within the organization are permitted to access that model. This ensures that only authorized personnel can use specific models based on your access control settings.

You can also give model access to a Virtual Account Read more here

Tryout your Integrated Models

Once you've added the providers and models through the integrations, you're ready to test them out.

Try in the Playground

To test your models in the playground, simply visit the playground interface and select the model from the available models dropdown. You can search for your models by the provider account name or model name, as shown below:

Try Programatically

TrueFoundry provides several options for testing models programmatically, including Curl, REST APIs, and platform libraries such as OpenAI, Langchain, and more. To access code snippets for integration:

1. Visit Gateway.
2. Select a model.
3. Click the "View Code" button to see auto-generated code snippets for different libraries, REST APIs, or simple Curl requests, as shown below.

To make a request, you'll need the BASE_URL and API_KEY. The BASE_URL is pre-filled in the auto-generated code snippets. For instructions on generating a new API_KEY, refer to the API Key Documentation.

Add LLMs from Anthropic, DeepSeek, Ollama and more

The process is pretty much the same. Follow these steps -

1. Navigate to Integrations.
2. Click on "Add New Integration".
3. Select the provider from the list.
4. Enter the required API keys and access credentials.

Once completed, your models will be ready for use in both the playground and your code.