Adding Models to LLM Gateway

Understanding the concept of a Provider and a Model -

What is a Provider?

A provider is a company or organisation that offers access to various language models. These providers offer different models, each with unique capabilities, performance characteristics, and pricing structures. Examples of popular providers include OpenAI and Anthropic.

What is a Model?

A model is a specific instance of an AI language model offered by a provider. Each model has distinct features and is trained on different datasets, leading to variations in performance and suitability for different tasks. For instance, OpenAI's GPT-3 and GPT-4 are different models, each with its own set of capabilities.

Steps to Add Providers and Models

This section captures how to add models along with setting up required access controls.

Access Control

In the above YAML example, everyone gets access to all the models. You can configure access control at the model level.

Adding access control through the UI

While adding a model as per the steps in the previous docs, you can configure the users and teams within the organisation who can access this model.

Add Self-hosted Models

You can also deploy an open-source or private model on TrueFoundry and add it to Gateway directly. This can also be extended to fine-tuned models and even embedding models.

Refer to our docs on Deploying LLMs From Model Catalog for how to deploy LLMs from our model catalogue in a few clicks.

Follow these steps once you have your model deployed: