This guide provides instructions for integrating DSPy with the Truefoundry AI Gateway.

​
What is DSPy?

DSPy is a framework for algorithmically optimizing language model prompts and weights. It provides a programming model for creating and optimizing LM-based systems through a declarative approach. With Truefoundry AI Gateway integration, you can make your DSPy pipelines production-ready with detailed insights on costs & performance metrics for each run, while accessing multiple LLM providers through a unified interface.

​
Prerequisites

Before integrating DSPy with TrueFoundry, ensure you have:
  1. TrueFoundry Account: Create a Truefoundry account with atleast one model provider and generate a Personal Access Token by following the instructions in Generating Tokens. For a quick setup guide, see our Gateway Quick Start
  2. DSPy Installation: Install DSPy using pip: pip install -U dspy

​
Setup Process

​
1. Generate Your TrueFoundry Access Token

Navigate to your TrueFoundry dashboard and generate a Personal Access Token:

​
2. Configure DSPy with TrueFoundry Gateway

DSPy integrates seamlessly with TrueFoundry’s gateway through the LM interface. Configure the LM with TrueFoundry’s gateway URL and your API key: 
import dspy

# Set up your TrueFoundry-enabled LM client
lm = dspy.LM(
    model="openai-main/gpt-4o",  # Use TrueFoundry model name
    api_key="your-truefoundry-api-key",
    api_base="your-truefoundry-gateway-url"
)

# Configure DSPy to use the TrueFoundry-enabled client
dspy.configure(lm=lm)
Replace:
  • your-truefoundry-api-key with your actual TrueFoundry API key
  • your-truefoundry-gateway-url with your TrueFoundry Gateway URL
  • openai-main/gpt-4o with your desired model from your TrueFoundry providers

​
3. Environment Variables Configuration

For persistent configuration across your DSPy applications, set these environment variables: 
export OPENAI_API_KEY="your-truefoundry-api-key"
export OPENAI_BASE_URL="your-truefoundry-gateway-url"

​
Usage Examples

​
Basic DSPy with TrueFoundry Gateway

Here’s a simple example demonstrating DSPy with TrueFoundry integration: 
import dspy

# Set up the TrueFoundry-enabled client
lm = dspy.LM(
    model="openai-main/gpt-4o",
    api_key="your-truefoundry-api-key",
    api_base="your-truefoundry-gateway-url"
)
dspy.configure(lm=lm)

# Simple completion
response = lm("Say this is a test!", temperature=0.7)
print(response)  # => ['This is a test!']

​
DSPy Signatures and Modules

Create more sophisticated DSPy programs using signatures and modules: 
import dspy

# Configure TrueFoundry LM
lm = dspy.LM(
    model="openai-main/gpt-4o",
    api_key="your-truefoundry-api-key",
    api_base="your-truefoundry-gateway-url"
)
dspy.configure(lm=lm)

# Define a signature for question answering
class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""
    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")

# Create a module using Chain of Thought
generate_answer = dspy.ChainOfThought(GenerateAnswer)

# Ask a question
question = "What is the capital of Brazil?"
pred = generate_answer(question=question)
print(f"Question: {question}")
print(f"Answer: {pred.answer}")
print(f"Reasoning: {pred.rationale}")

​
Advanced RAG System with DSPy

Build a complete RAG (Retrieval-Augmented Generation) system: 
import dspy

# Configure TrueFoundry-enabled LM
lm = dspy.LM(
    model="openai-main/gpt-4o",
    api_key="your-truefoundry-api-key",
    api_base="your-truefoundry-gateway-url"
)
dspy.configure(lm=lm)

# Define signatures for RAG pipeline
class GenerateSearchQuery(dspy.Signature):
    """Write a simple search query that will help answer a complex question."""
    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    query = dspy.OutputField()

class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""
    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")

# Mock retrieval function (replace with your actual retrieval system)
def retrieve(query: str) -> list[str]:
    # This would typically connect to your vector database or search API
    return [
        "Brazil is a country in South America.",
        "The capital of Brazil is Brasília.",
        "Brasília was founded in 1960 and became the capital."
    ]

# Create RAG module
class RAG(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_query = dspy.ChainOfThought(GenerateSearchQuery)
        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)
    
    def forward(self, question):
        # Generate search query
        search_query = self.generate_query(context="", question=question).query
        
        # Retrieve relevant documents
        passages = retrieve(search_query)
        context = "\n".join(passages)
        
        # Generate final answer
        pred = self.generate_answer(context=context, question=question)
        return dspy.Prediction(context=context, answer=pred.answer)

# Use the RAG system
rag = RAG()
question = "What is the capital of Brazil?"
result = rag(question)

print(f"Question: {question}")
print(f"Context: {result.context}")
print(f"Answer: {result.answer}")

​
DSPy Optimization with TrueFoundry

Optimize your DSPy programs using the built-in optimizers: 
import dspy
from dspy.evaluate import Evaluate

# Configure TrueFoundry LM
lm = dspy.LM(
    model="openai-main/gpt-4o",
    api_key="your-truefoundry-api-key",
    api_base="your-truefoundry-gateway-url"
)
dspy.configure(lm=lm)

# Define a simple QA module
class QA(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_answer = dspy.ChainOfThought("question -> answer")
    
    def forward(self, question):
        prediction = self.generate_answer(question=question)
        return dspy.Prediction(answer=prediction.answer)

# Create training examples
trainset = [
    dspy.Example(question="What is the capital of France?", answer="Paris"),
    dspy.Example(question="What is the capital of Japan?", answer="Tokyo"),
    dspy.Example(question="What is the capital of Brazil?", answer="Brasília"),
]

# Define evaluation metric
def validate_answer(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

# Initialize module and optimizer
qa_module = QA()
optimizer = dspy.BootstrapFewShot(metric=validate_answer)

# Optimize the module
optimized_qa = optimizer.compile(qa_module, trainset=trainset)

# Test the optimized module
question = "What is the capital of Brazil?"
result = optimized_qa(question)
print(f"Optimized Answer: {result.answer}")

​
Benefits of Using TrueFoundry Gateway with DSPy

  1. Cost Tracking: Monitor and track costs across all your DSPy operations with detailed metrics
  2. Security: Enhanced security with centralized API key management
  3. Access Controls: Implement fine-grained access controls for different teams
  4. Rate Limiting: Prevent API quota exhaustion with intelligent rate limiting
  5. Fallback Support: Automatic failover to alternative providers when needed
  6. Analytics: Detailed analytics and monitoring for all LLM calls in your DSPy pipelines
  7. Multi-Provider Support: Seamlessly switch between different model providers (OpenAI, Anthropic, Google, etc.)
  8. Performance Optimization: Track and optimize the performance of your DSPy modules