Intro to LLM Gateway

An LLM Gateway provides a unified interface to manage your organization's LLM usage. Below are the key features that enhance functionality and security:

• Unified API: Access multiple LLM providers through a single OpenAI compatible interface, requiring no code changes.

• API Key Security: Secure and centralized management of credentials.

• Governance & Control: Set limits, enforce access controls, and apply content filtering to manage how LLMs are used within your organization.

• Rate Limiting: Implement measures to prevent abuse and ensure fair usage across users.

• Observability: Track and analyse usage, costs, latency, and overall performance.

• Cost Management: Monitor spending and configure budget alerts to keep expenses under control.

• Audit Trails: Maintain logs of all interactions with LLMs to support compliance and auditing requirements.

LLM Playground

The LLM Playground is a UI for the LLM Gateway where you can interact with the various models you've added. Below is an overview of the features:

1. Support for multiple model types
   1. Chat Models
   2. Completion Models
   3. Embedding Models
2. Image Upload: Upload images for image captioning or visual question answering. This is only available for models that support images such as GPT-4o
3. Model Comparison: Compare responses from different completion models to evaluate their performance.
4. System Prompts: Use predefined system prompts to guide model behaviour.System prompt inform how the model should respond. Sample system prompt - Be clear, concise, and polite in your responses. Avoid sharing any sensitive or personal information.

Speed features of LLM Gateway

• Near-Zero Overhead: Just 3-5 ms added latency
• Optimised Backend: Built with performant Node.js framework and optimised config caching
• Smart Routing: Minimal processing overhead
• Edge Ready: Deploy close to your apps