!(function () {
  var e, t, n;
  (e = "7dcab2ef5bdce24"),
    (t = function () {
      Reo.init({ clientID: "7dcab2ef5bdce24" });
    }),
    ((n = document.createElement("script")).src =
      "https://static.reo.dev/" + e + "/reo.js"),
    (n.defer = !0),
    (n.onload = t),
    document.head.appendChild(n);
})();


GenericInfra Requirements

Existing Cluster

Requirements for Truefoundry installation on Generic

Requirements

truefoundry Docs

Setup Your Account

Key Concepts

Setup For CLI

Introduction

Launch Jupyter Notebook

Launch a SSH Server

Introduction To A Service

Interacting With Your Service

Update, Rollback, Promote

Introduction To A Job

Interacting With Your Job

Monitor Your Job

Powered by Flyte (An OpenSource Workflow Orchestrator)

Introduction To Workflow

Creating Your First Workflow

Interacting With Workflow

Workflow Concepts

Introduction to Async Service

Deploy your first Async Service

Monitor Your Async Service

Introduction to Volume

Creating And Utilizing Volume

Interacting With Your Volume

Creating Statically Provisioned Volumes

Introduction To ML Repo

Create Tracing Project

Introduction To Secrets

Manage Secrets

Collaboration And Access Control

Personal Access Tokens and Virtual Accounts

Generating TrueFoundry API Keys

Create Custom K8s Objects

Store deployment configuration in Git and implement a approval process for changes

Setup Gitops Using Truefoundry

Applying Custom Policies

Using TrueFoundry Secrets In Integrations

Understanding modes of deployment in truefoundry

Modes Of Deployment

Deploy TrueFoundry In An Air Gapped Environment

Migrate Pytorch Sagemaker Endpoints To TrueFoundry Platform

Guide To Inference Model On Truefoundry

Quick Start

Control access of models among teams, users and applications

Access Control

Rate Limiting

Fallback a request to another model on failure from one model

Fallback

Analytics

Metadata

Export Logs/Traces

Architecture

Deploy AI Gateway Only

Authentication for AI Gateway

Request and Response Headers

Generate chat-based completions using the specified model.

Chat Completions

Generate embeddings for the given input using the specified model.

Generate Embeddings

Rerank documents based on the given query and parameters.

Rerank Documents

List input items for a specific model response

List Model Response Input Items

Get Model Response

Delete Model Response

Generate model responses using the specified model.

Model Responses

Transcribes audio into the input language.

Transcribe Audio

List batches

Create Batch

Get information about a specific batch process

Get Batch

Get Batch Output

Cancel Batch

List Files

Upload File

Get File

Delete file

Get File Content

Post moderations

Deploying LLMs

Deploy optimized TensorRT-LLM Engines using NIM Containers

Deploying NVIDIA NIM Models

Measure token generation throughput, Time to First Token (TTFT), Inter Token Latency of LLMs via the chat completions API

Benchmarking LLMs

Finetune Llama, Mistral, Mixtral and more on one or more GPUs

Finetuning LLMs

Prompt Management

Overview

Tracing

Tracing In CrewAI

Tracing In LangGraph

Tracing In Agno

Tracing FastAPI applications

Retrieves a list of all latest Clusters. Pagination is available based on query parameters.

List Clusters

Create or Update cluster with provided manifest

Create or Update Cluster

Get Cluster

Delete cluster associated with provided cluster id

Delete cluster

List addons for the provided cluster.Pagination is available based on query parameters.

List Addons

Get Cluster status

List workspaces associated with the user. Optional filters include clusterId, fqn, and workspace name. Pagination is available based on query parameters.

List Workspace

Creates a new workspace or updates an existing one based on the provided manifest.

Create or Update Workspace

Get workspace associated with provided workspace id

Get Workspace

Deletes the workspace with the given workspace ID.
    - Removes the associated namespace from the cluster.
    - Deletes the corresponding authorization entry.

Delete Workspace

Creates or updates an MLRepo entity based on the provided manifest.

Create or Update MLRepo

Get a ml repo by id
Args:
    id: Unique identifier of the ml repo to get
    user_info: Authenticated user information

Returns:
    GetMLRepoResponse: The ml repo

Get Ml Repo

Delete a ml repo
Args:
    id: Unique identifier of the ml repo to delete
    user_info: Authenticated user information

Returns:
    EmptyResponse: Empty response indicating successful deletion

Delete Ml Repo

List ml repos
Args:
    filters: Filters for the ml repos
    user_info: Authenticated user information

Returns:
    ListMLReposResponse: List of ml repos

List Ml Repos

Retrieves a list of all latest applications. Supports filtering by application ID, name, type, and other parameters. Pagination is available based on query parameters.

List Applications

Create a new Application Deployment based on the provided manifest.

Create Application Deployment

Get Application associated with the provided application ID.

Get Application

Delete Application associated with the provided application ID.

Delete Application

Fetch all deployments for a given application ID with optional filters such as deployment ID or version. Supports pagination.

List Application Deployments

Get Deployment associated with the provided application ID and deployment ID.

Get Deployment details

Pause a running application by scaling to 0 replicas

Pause an Application

Resume a paused application by scaling back to the original number of replicas

Resume an Application

Cancel an ongoing deployment associated with the provided application ID and deployment ID.

Cancel an Ongoing Deployment

List Job Runs for provided Job Id. Filter the data based on parameters passed in the query

List Job Runs

Get Job Run for provided jobRunName and jobId

Get Job Run

Delete Job Run for provided jobRunName and jobId

Delete Job Run

Trigger Job for provided deploymentId or applicationId

Trigger Job

Terminate Job for provided deploymentId and jobRunName

Terminate Job

Get Model

Delete Model

List Models

Get Model Version

Delete Model Version

List Model Versions

Apply Model Version

Get Artifact

Delete Artifact

List Artifacts

Get Artifact Version

Delete Artifact Version

List Artifact Versions

Apply Artifact Version

Get Signed Urls For Artifact Version

Create Multi Part Upload

Stage Artifact Version

List Files For Artifact Version

Mark Stage Failure

Get Prompt

Delete Prompt

List Prompts

Get Prompt Version

Delete Prompt Version

List Prompt Versions

Apply Prompt Version

List the secret groups associated with a user along with the associated secrets for each group. Filtered with the options passed in the query fields. Note: This method does not return the secret values of the associatedSecrets in the response. A separate API call to `/v1/secrets/{id}` should be made to fetch the associated secret value.

List Secret Groups

Creates a secret group with secrets in it. A secret version for each of the created secret is created with version number as 1. The returned secret group does not have any secret values in the associatedSecrets field. A separate API call to `/v1/secrets/{id}` should be made to fetch the associated secret value.

Create Secret Group

Get Secret Group by id. This method does not return the secret values of the associatedSecrets in the response. A separate API call to `/v1/secrets/{id}` should be made to fetch the associated secret value.

Get Secret Group

Updates the secrets in a secret group with new values. A new secret version is created for every secret that has a modified value and any omitted secrets are deleted. The returned updated secret group does not have any secret values in the associatedSecrets field. A separate API call to `/v1/secrets/{id}` should be made to fetch the associated secret value.

Update Secret Group

Deletes the secret group, its associated secrets and secret versions of those secrets.

Delete Secret Group

List secrets associated with a user filtered with optional parameters passed in the body.

List Secrets

Get Secret associated with provided id. The secret value is not returned if the control plane has `DISABLE_SECRET_VALUE_VIEW` set

Get Secret

Deletes a secret and its versions along with its values.

Delete Secret

List the active deployments that are associated with a secret.

List Associated Active Deployments

Fetches deployment specifications for a model version or a HuggingFace model URL.

Get Deployment Specifications

Fetches deployment specifications for a NIM Model

Requirements	Description	Reason for requirement
Kubernetes Version	Kubernetes version 1.30 or higher	Required for latest security features and compatibility
Worker Nodes	- Minimum 3 worker nodes - Each worker node: 4 vCPUs, 16GB RAM - For GPU workloads: NVIDIA GPU-enabled nodes	Required for running core Truefoundry components and user workloads
Block storage	Block storage for the worker nodes and persistent volumes	Required to ensure that the worker nodes and deployments have enough storage to run the workloads. Min 100GB per worker node and persistent volumes as needed
NFS storage	NFS storage for artifact cache	Required to ensure that the ML deployment artifacts have enough storage to run
Egress access For Docker Registry	- `public.ecr.aws` - `quay.io` - `ghcr.io` - `docker.io/truefoundrycloud` - `docker.io/natsio` - `nvcr.io` - `registry.k8s.io`	This is to download docker images for Truefoundry, ArgoCD, NATS, GPU operator, ArgoRollouts, ArgoWorkflows, Istio, Keda.
Load Balancer	Load Balancer for the services	Required to ensure that the services are accessible from the internet if required to exposed. Load balancer should get pre-filled with the IP address given by address pool
DNS	Domain for service endpoints	Examples: `.internal.example.com`, `.external.example.com`, `tfy.example.com`. Wildcard preferred for developer service deployments
Certificate	Certificate for the domains	Required for terminating TLS traffic to the services. Can be managed through cert-manager or custom certificates. Check here for more details.

Getting Started

Train and Deploy Models

Service Deployment

Job Deployment

Workflow Deployment

Async Service Deployment

Volumes

ML Repository

Secret Management

Access Control

Platform

Deploying On Your Own Cloud

HOW TOs

Requirements

GenericInfra Requirements

Existing Cluster

Getting Started

Train and Deploy Models

Service Deployment

Job Deployment

Workflow Deployment

Async Service Deployment

Volumes

ML Repository

Secret Management

Access Control

Platform

Deploying On Your Own Cloud

HOW TOs

​GenericInfra Requirements

​Existing Cluster

GenericInfra Requirements

Existing Cluster