Truefoundry Docs

TrueFoundry AI Gateway supports multimodal inputs including images, audio, and video, allowing you to work with different data formats in your chat completions.

Supported Input Types

Images

Send images as part of your chat completion requests using either URLs or base64 encoding:

Using Image URLs

from openai import OpenAI

client = OpenAI(
    api_key="your_truefoundry_api_key",
    base_url="<truefoundry-base-url>/api/llm/api/inference/openai"
)

response = client.chat.completions.create(
    model="openai-main/gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/image.jpg"
                    }
                }
            ]
        }
    ]
)

Using Base64 Encoded Images

import base64

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

response = client.chat.completions.create(
    model="openai-main/gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{encode_image('image.jpeg')}"
                    }
                }
            ]
        }
    ]
)

Audio

Send audio files in supported formats (MP3, WAV, etc.). Currently supported for Google Gemini models:

Using Audio URLs

response = client.chat.completions.create(
    model="internal-google/gemini-2-0-flash",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/audio.wav",
                        "mime_type": "audio/wav" # required for gemini models
                    }
                }
            ]
        }
    ]
)

Using Base64 Encoded Audio

import base64

def encode_audio(audio_path):
    with open(audio_path, "rb") as audio_file:
        return base64.b64encode(audio_file.read()).decode('utf-8')

response = client.chat.completions.create(
    model="internal-google/gemini-2-0-flash",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:audio/wav;base64,{encode_audio('audio.wav')}"
                    }
                }
            ]
        }
    ]
)

Video

Video processing is natively supported for Google Gemini models:

Using Video URLs

response = client.chat.completions.create(
    model="internal-google/gemini-2-0-flash",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what's happening in this video"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://www.youtube.com/watch?v=example",
                        "mime_type": "video/mp4" # required for gemini models
                    }
                }
            ]
        }
    ]
)

Using Base64 Encoded Video

import base64

def encode_video(video_path):
    with open(video_path, "rb") as video_file:
        return base64.b64encode(video_file.read()).decode('utf-8')

response = client.chat.completions.create(
    model="internal-google/gemini-2-0-flash",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what's happening in this video"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:video/mp4;base64,{encode_video('video.mp4')}",
                        "mime_type": "video/mp4" # required for gemini models
                    }
                }
            ]
        }
    ]
)

Supported Models

Input Type	Supported Models
Images	GPT-4o, GPT-4 Vision, Claude 3, Gemini Pro Vision
Audio	Google Gemini models (Gemini 2.0 Flash, etc.)
Video	Google Gemini models (Gemini 2.0 Flash, etc.)

Always check model capabilities before sending multimodal inputs to avoid errors.

Get Started

Developer Guide

MCP Registry and Gateway

Observability

Integrations

Deployment

API Reference

Chat

Agent

MCP

Embeddings

Rerank

Responses

Image

Audio

Batch

Files

Moderations

Models

Multimodal Inputs

Supported Input Types

Using Image URLs

Using Base64 Encoded Images

Using Audio URLs

Using Base64 Encoded Audio

Using Video URLs

Using Base64 Encoded Video

Supported Models

Get Started

Developer Guide

MCP Registry and Gateway

Observability

Integrations

Deployment

API Reference

Chat

Agent

MCP

Embeddings

Rerank

Responses

Image

Audio

Batch

Files

Moderations

Models

​Supported Input Types

​Supported Models

Supported Input Types

Supported Models