All the components in Truefoundry are modular and you can decide to use only the components you need. For e.g., if you don’t need the AI engineering module, you can just use the AI gateway module.
AI Engineering
AI engineering module primarily enables datascientists to deploy their models, agents and workflows on your own infrastructure while providing a single place to manage all the AI assets. It abstracts out the underlying infrastructure to enable rapid experimentation and deployment, while making sure it adheres to the guardrails and principles set by the organization.Truefoundry doesn’t provide compute. You bring your own cloud account or on-prem hardware. Truefoundry will connect with it and enable you to deploy your models, agents and workflows.
All the models and artifacts are also stored on your own storage.
Jupyter Notebooks / Remote SSH
Start Jupyter Notebooks or connect your existing IDE to remote compute on any cloud/on-prem hardware including GPUs.
Train Models / Batch Inference
Use the Jobs feature in Truefoundry to run training or batch inference jobs either manually or on a schedule.
Model Registry
Store and Version your models and artifacts.
Model Inference
Deploy your models in any framework (Transformers, PyTorch, TensorFlow, SkLearn, XGBoost, etc) as realtime APIs.
Workflows
Deploy and monitor complex ML pipelines.
Service Deployment
Deploy any service (REST, gRPC, etc) or your Streamlit, Gradio, Flask, FastAPI applications.
LLM Deployment
Deploy LLMs from HuggingFace/own model registry using vLLM / Sglang /TRTLLM with low latency, high througput and faster autoscaling.
LLM Finetuning
Finetune LLM models on your own data from the model catalogue or any HuggingFace LLM model.
Async Inference
Deploy async inference services backed by queue of your choice to handle inference with higher latency.
AI Gateway
The AI gateway provides a single interface to access all the LLMs and AI models, MCP servers and agents within an organization. It comes with access control, key management, governance and monitoring inbuilt which enables developers to focus on building great applications without worrying about the underlying models, keys, observability and platform teams to impose rate and budget limits, access control, audit and security guardrails.LLM Gateway
Call 1000+ LLM models using a single API.
MCP Registry
Deploy MCP servers from the model catalogue.
MCP Gateway
Deploy MCP servers from the model catalogue.
Prompt Management
Create, store and version prompts and use them via the Gateway.
Tracing and Observability
Trace and monitor all requests across LLMs and MCP servers going through the Gateway.
Agent Gateway
Register all agents in the Gateway and call them via a single endpoint. Coming Soon.