Imagine you're working on a project that involves handling sensitive or proprietary information. Due to security and privacy concerns, you decide to use an Open Source LLM instead of relying on a cloud-based service like OpenAI. This decision allows you to have full control over the model and the data it processes, ensuring that everything remains within your own infrastructure.

Now, you start experimenting with different Open Source LLMs to determine which one aligns best with your project requirements. Here you encounter a common challenge. Setting up and managing these models on your infrastructure requires significant effort. You need to handle deployment, fine-tuning, and various other aspects, making the experimentation phase labor-intensive.

This is where TrueFoundry's LLM Gateway steps in, offering a hosted solution for popular Open Source LLMs. By eliminating the need for self-hosting, LLM Gateway streamlines the experimentation phase and empowers you with:

• Ease of Experimentation: LLM Gateway allows you to experiment with different Open Source LLMs without the hassle of setting up and managing them on your own infrastructure.
• Comparison Capabilities: The platform allows you to compare up to four models simultaneously, streamlining the process of finding the right fit for your specific language model needs.
• User-Friendly Interface: The platform offers a user-friendly interface, simplifying the experimentation process.
• Metric Tracking: LLM Gateway provides metric tracking, enabling you to assess the performance of each model during your experimentation phase.
• OpenAI Integration: If your experimentation involves comparing Open Source LLMs with a well-established model like OpenAI, LLM Gateway facilitates this integration.

Accessing the Playground

To access the Truefoundry LLM Playground, follow these simple steps:

1. Login to TrueFoundry and enter the platform
2. Navigate to the "Models" tab located in the left panel of your Truefoundry account interface.
3. Click on the "Model Catalogue" at the top left corner.
4. Look for the "Open Playground" button and give it a click.

Voila! The playground should now be open to you!

Setting up the Playground

📘

Note:

The "Settings" tab is exclusively accessible to tenant administrators. If you cannot view this tab and do not have the ability to utilize models that rely on API access (such as OpenAI, Anthropic, or Cohere), or access certain public models, please reach out to your tenant administrator for assistance.

To configure the playground and control the LLMs that are displayed in the catalog for your tenant members, you can follow these steps:

1. Navigate to the "Settings" tab, which is accessible through the top bar.
2. Within the "Settings" tab, you can choose which models should be available to members by enabling or disabling the checkboxes located under the "Model Selection" section.
3. Additionally, you have the option to provide the API key for accessing LLMs offered by OpenAI, Cohere, and Anthropic.

Interacting with the Playground

Upon entering the Truefoundry LLM Playground, you'll be greeted with a text box that eagerly awaits your prompts. But hold on – before you jump into the world of text generation, you need to select a model and set your preferences.

1. Start by clicking on the settings button located at the top right corner of the playground interface.

2. This will open up a settings panel where you can fine-tune the parameters to suit your experiment's needs.

Model Selection and Settings

In the settings panel, you'll find several key parameters that allow you to customize your model's behavior:

• Available Models Dropdown: This dropdown presents a wide array of options, including OpenAI models, Public Open Source models, and our own Truefoundry-hosted open-source models. Select the one that piques your interest.
• Predefined Settings: Each model comes with a set of predefined settings, handpicked by our skilled engineers to provide optimal performance. However, feel free to tweak these settings to experiment and achieve your desired results.
• Parameters to Tweak:
  • Maximum Length: Control the length of the generated output.
  • Temperature: Adjust the randomness of the output.
  • Top P: Limit the diversity of choices during text generation.
  • Top K: Further refine the diversity by controlling the number of choices considered.
  • Repetition Penalty: Mitigate repeated phrases in the generated text.
  • Stop Sequences: Specify sequences that, when generated, signal the model to stop.

Let the Experimentation Begin!

With your model selected and settings configured, you're ready to Start your experimentation. Click the 'X' button in the settings panel to close it, and now you can start.

Enter prompts, questions, or topics, and watch your selected LLM generating responses for this prompts.

Examples:

• GPT-4

• Llama-2-70b-chat

Programmatic Interaction

For those who prefer to tinker with code, the playground offers a programmatic approach:

1. Click on the code snippet icon to access model-related code snippets.

2. These code snippets come pre-configured with the chosen model and parameters.

3. Remember to replace with your own API Key. If you're unsure how to generate your API Key, refer to the following documentation for step-by-step instructions.

Prepare to be amazed as you interact with language in ways you never thought possible. The Truefoundry LLM Playground empowers you to experiment, learn, and push the boundaries of AI-driven linguistic creativity. So go ahead, dive in, and let your words come to life in extraordinary ways!

Effortless Model Comparison with Truefoundry PlayGround

Explore the fascinating world of language models with a new dimension – comparison. Truefoundry PlayGround introduces an intuitive Compare feature that allows you to assess different models side by side, helping you make informed decisions for your specific needs. To do this you can:

1. Navigate to the "Compare" tab within the PlayGround interface.
2. Click on the settings icon to access the comparison settings.

3. From the list of available models, select the ones you wish to compare, by clicking on the checkbox besides the model.

Example:

For example lets compare llama-2-70b-chat and vicuna-13b

• llama-2-70b-chat

• vicuna-13b