This guide provides instructions for integrating Jan AI with the Truefoundry AI Gateway.

​
What is Jan AI?

Jan is an open source ChatGPT-alternative that runs 100% offline on your computer. It’s a desktop application designed for privacy-conscious users who want to chat with AI without sending their data to external servers. Jan can run powerful local models like Llama3, Gemma, and Mistral directly on your device, while also providing the flexibility to connect to cloud AI providers when needed.

​
Key Features of Jan AI

  • 100% Offline Operation: Run AI models locally on your computer without internet dependency, ensuring complete privacy and data security
  • Model Hub: Download and run powerful open-source models like Llama3, Gemma, and Mistral directly on your device
  • Cloud AI Integration: Connect to cloud models from providers like OpenAI, Groq, and Cohere when you need more computational power
  • Local API Server: Set up and run your own OpenAI-compatible API server using local models with just one click
  • Chat with Files: Talk to PDFs, notes, and other documents directly to get summaries, answers, or insights
  • Fully Customizable: Customize Jan with extensions and adjust alignment, moderation, and censorship levels to your needs

​
Prerequisites

Before integrating Jan AI with TrueFoundry, ensure you have:
  1. TrueFoundry Account: Create a Truefoundry account and follow our Gateway Quick Start
  2. Jan AI Installation: Download and install Jan from jan.ai (3.9M+ downloads, free & open source)

​
Setup Process

​
1. Configure Jan AI with TrueFoundry

You will get your ‘truefoundry-api-key’, ‘truefoundry-gateway-url’ and model name directly from the unified code snippet

​
2. Configure OpenAI Provider in Jan Settings

  1. Open Jan AI Application
  2. Navigate to Settings → Model Providers → OpenAI
Jan AI settings showing OpenAI provider configuration

Jan AI Settings - OpenAI Provider Configuration

  1. Update the configuration:
    • API Key: Enter your TrueFoundry API key
    • Base URL: Enter your TrueFoundry gateway URL (e.g., https://your-truefoundry-gateway.com/api/llm)

​
3. Add TrueFoundry Models

  1. In the Models section, click the ”+” button to add new models
  2. Enter model IDs in TrueFoundry format (e.g., openai-main/gpt-4o, anthropic-main/claude-3-5-sonnet-20241022)
Jan AI add new model dialog showing TrueFoundry model format

Add New Model Dialog

​
Usage Examples

​
Start Chatting with TrueFoundry Models

Once configured, you can start using TrueFoundry models in Jan’s chat interface:
Jan AI chat interface showing TrueFoundry models in use

Jan AI Chat Interface with TrueFoundry Models

  1. Select Model: Choose your configured TrueFoundry model from the model dropdown (e.g., openai-main/gpt-4o)
  2. Start Chatting: Begin conversations with the AI - requests will be routed through TrueFoundry’s gateway
  3. Switch Models: Easily switch between local models and TrueFoundry cloud models as needed

​
Observability and Governance

Monitor your Jan AI usage through TrueFoundry’s metrics tab: TrueFoundry metrics With TrueFoundry’s AI gateway, you can monitor and analyze:
  • Performance Metrics: Track key latency metrics like Request Latency, Time to First Token (TTFS), and Inter-Token Latency (ITL) with P99, P90, and P50 percentiles
  • Cost and Token Usage: Gain visibility into your application’s costs with detailed breakdowns of input/output tokens and the associated expenses for each model
  • Usage Patterns: Understand how your application is being used with detailed analytics on user activity, model distribution, and team-based usage
  • Hybrid Usage: Monitor the balance between local model usage and cloud model requests through TrueFoundry
  • Rate limit and Load balancing: Set up rate limiting, load balancing and fallback for your models