Skip to main content
This guide provides instructions for integrating Langflow with the Truefoundry AI Gateway.

​
What is Langflow?

Langflow is a visual framework for building multi-agent and RAG applications. It’s a low-code app builder for RAG and multi-agent AI applications, providing a Python-based platform that allows users to create flows using a drag-and-drop interface or through code.

​
Key Features of Langflow

  1. Visual Drag-and-Drop Interface: Langflow provides an intuitive visual interface where users can build complex AI workflows by simply dragging and dropping components. This eliminates the need for extensive coding and makes AI application development accessible to non-technical users and developers alike.
  2. Multi-Agent and RAG Support: Built-in support for Retrieval-Augmented Generation (RAG) and multi-agent architectures allows users to create sophisticated AI applications that can access external knowledge bases and coordinate multiple AI agents to solve complex tasks collaboratively.
  3. Code and No-Code Flexibility: Langflow offers both visual workflow creation and Python SDK integration, allowing users to switch between drag-and-drop interfaces and programmatic control. This flexibility enables both rapid prototyping and production-ready deployments with custom logic.

​
Prerequisites

Before integrating Langflow with TrueFoundry, ensure you have:
  1. TrueFoundry Account: Create a Truefoundry account with atleast one model provider and generate a Personal Access Token by following the instructions in Generating Tokens. For a quick setup guide, see our Gateway Quick Start
  2. Langflow Installation: Install Langflow using either the Python package or Docker deployment
  3. Load Balance Configuration: Setup load balancing configuration for your desired models (see Setup Process section below)

​
Why Load Balancing Configuration is Necessary

Langflow works optimally with standard OpenAI model names (like gpt-4, gpt-4o-mini), but may experience compatibility issues with Truefoundry’s fully qualified model names (like openai-main/gpt-4 or azure-openai/gpt-4). When Langflow encounters these fully qualified model names directly, it may not function as expected due to internal processing differences. The Solution: Load balancing configuration allows you to:
  1. Use standard model names in your Langflow configurations (e.g., gpt-4)
  2. Have Truefoundry Gateway automatically route the request to the fully qualified target model (e.g., openai-main/gpt-4)
This approach ensures seamless compatibility while still allowing you to access any model through the Truefoundry Gateway.

​
Setup Process

​
1. Generate Your TrueFoundry Access Token

Navigate to your TrueFoundry dashboard and generate a Personal Access Token:
TrueFoundry dashboard showing Personal Access Token creation interface

​
2. Setup Load Balance Configuration

Create a load balancing configuration to route your requests to specific model providers: 
name: langflow-loadbalancing-config
type: gateway-load-balancing-config
rules:
  - id: langflow-load-balancing
    type: weight-based-routing
    when:
      models:
        - gpt-4o
    load_balance_targets:
      - target: openai-main/gpt-4o
        weight: 100
This configuration ensures that when you request standard model names through Langflow, your requests will be routed to the appropriate models via the Truefoundry Gateway.

​
3. Configure Langflow Language Model Component

In your Langflow interface, configure the Language Model component with TrueFoundry Gateway settings:
Langflow interface showing Language Model component configuration panel
  1. Model Provider: Select “OpenAI” from the dropdown
  2. Model Name: Use the standard model name from your load balancing configuration (e.g., gpt-4o-mini)
  3. OpenAI API Key: Enter your TrueFoundry Personal Access Token
  4. OpenAI API Base: Set to https://{controlPlaneUrl}/api/llm/api/inference/openai
TrueFoundry playground showing unified code snippet with base URL and model name for Langflow integration
Langflow OpenAI component configuration with TrueFoundry API settings
Replace {controlPlaneUrl} with your actual Truefoundry control plane URL.

​
4. Advanced Configuration in Agent Settings

For more advanced flows using agents, configure the agent component with TrueFoundry settings: Ensure the following settings are configured:
  • Model Name: Use the load-balanced model name (e.g., gpt-4)
  • OpenAI API Base: https://{controlPlaneUrl}/api/llm/api/inference/openai
  • OpenAI API Key: Your TrueFoundry Personal Access Token

​
Usage Examples

​
Basic Chat Flow

Create a simple chat flow using the configured Language Model:
Langflow canvas showing a basic chat flow with Chat Input, OpenAI model, and Chat Output components connected
  1. Drag and drop a “Chat Input” component
  2. Connect it to your configured “OpenAI” Language Model component
  3. Connect the output to a “Chat Output” component
  4. Run the flow to test the integration

​
Multi-Agent RAG Application

For more complex applications involving RAG and multiple agents: 
# Example of using Langflow with TrueFoundry Gateway programmatically
from langflow import load_flow_from_json

# Load your flow configuration
flow = load_flow_from_json("path/to/your/flow.json")

# The flow will automatically use the configured TrueFoundry Gateway
# for all OpenAI model calls
result = flow.run(
    input_value="What is the capital of Brazil?",
    tweaks={
        "OpenAI-model_name": "gpt-4",  # This will be load-balanced
        "OpenAI-openai_api_base": "https://{controlPlaneUrl}/api/llm/api/inference/openai",
        "OpenAI-openai_api_key": "your-truefoundry-api-key"
    }
)

​
Environment Variables Configuration

Alternatively, you can set environment variables for easier configuration across multiple flows: 
export OPENAI_API_KEY="your-truefoundry-api-key"
export OPENAI_API_BASE="https://{controlPlaneUrl}/api/llm/api/inference/openai"

​
Understanding Load Balancing

When you use Langflow with standard model names like gpt-4, your requests get load-balanced according to your configuration. In the example configuration above, any request to gpt-4 will be routed to openai-main/gpt-4 with 100% of the traffic. You can create more sophisticated routing rules with multiple targets and different weights: 
load_balance_targets:
  - target: openai-main/gpt-4
    weight: 70
  - target: azure-openai/gpt-4
    weight: 30
With this configuration, approximately 70% of your requests would go to OpenAI’s GPT-4 model, and 30% would go to Azure OpenAI GPT-4 model.

​
Benefits of Using TrueFoundry Gateway with Langflow

  1. Cost Tracking: Monitor and track costs across all your Langflow applications
  2. Security: Enhanced security with centralized API key management
  3. Access Controls: Implement fine-grained access controls for different teams
  4. Rate Limiting: Prevent API quota exhaustion with intelligent rate limiting
  5. Fallback Support: Automatic failover to alternative providers when needed
  6. Analytics: Detailed analytics and monitoring for all LLM calls
⌘I