GCP Architecture Diagram

Access Policies

This guide describes the architecture diagram and access policies in a compute plane in your GCP account

Architecture

truefoundry Docs

About TrueFoundry

Create And Setup Your Account

Key Concepts and Information Architecture to get started

TrueFoundry Key Concepts

Create Your First Deployment

Setup For CLI

Personal Access Tokens and Virtual Accounts

Generating TrueFoundry API Keys

Introduction To A Service

Interacting With Your Service

Update, Rollback, Promote

Introduction To A Job

Interacting With Your Job

Monitor Your Job

Introduction To ML Repo

Quick Start

Powered by Flyte (An OpenSource Workflow Orchestrator)

Introduction To Workflow

Creating Your First Workflow

Interacting With Workflow

Workflow Concepts

Infra Set Up For Workflows

Introduction to LLMOps

Deploying LLMs

Deploy optimized TensorRT-LLM Engines using NIM Containers

Deploying NVIDIA NIM Models

Deploy LLMs using Nvidia TensorRT LLM (TRT LLM)

Finetune Llama, Mistral, Mixtral and more on one or more GPUs

Finetuning LLMs

Measure token generation throughput, Time to First Token (TTFT), Inter Token Latency of LLMs via the chat completions API

Benchmarking LLMs

Launch Jupyter Notebook

Launch a SSH Server

Introduction to Volume

Creating And Utilizing Volume

Interacting With Your Volume

Creating Statically Provisioned Volumes

Introduction to Async Service

Deploy your first Async Service

Monitor Your Async Service

Introduction To Secrets

Manage Secrets

Collaboration And Access Control

Understanding modes of deployment in truefoundry

Modes Of Deployment

Deploy Truefoundry In An Air Gapped Environment

Migrate Pytorch Sagemaker Endpoints To TrueFoundry Platform

Create Custom K8s Objects

Store deployment configuration in Git and implement a approval process for changes

Setup Gitops Using Truefoundry

Using Images From NVIDIA NGC Container Registry

Using TrueFoundry Secrets In Integrations

Service

AsyncService

Helm

Image And Build

Deployment Additional Configuration

Guide To Inference Model On Truefoundry

Intro to AI Gateway

OpenAI, Anthropic, DeepSeek and More

AWS Bedrock

Google Vertex

Azure OpenAI

Reasoning Models (Claude Only)

Realtime Models

Self Hosted Models

Batch Predictions with TrueFoundry AI Gateway

Databricks Models

LLM Playground

Integrate Gateway to your Apps

Control access of models among teams, users and applications

Access Control

Rate Limiting

Canary Testing

Fallback a request to another model on failure from one model

Fallback

Analytics

Metadata

Export Logs/Traces

Prompt Management

Retrieves a list of all latest Clusters. Pagination is available based on query parameters.

List Clusters

Create or Update cluster with provided manifest

Create or Update Cluster

Get Cluster

Delete cluster associated with provided cluster id

Delete cluster

List addons for the provided cluster.Pagination is available based on query parameters.

List Addons

Get Cluster status

List workspaces associated with the user. Optional filters include clusterId, fqn, and workspace name. Pagination is available based on query parameters.

List Workspace

Creates a new workspace or updates an existing one based on the provided manifest.

Create or Update Workspace

Get workspace associated with provided workspace id

Get Workspace

Deletes the workspace with the given workspace ID.
    - Removes the associated namespace from the cluster.
    - Deletes the corresponding authorization entry.

Delete Workspace

Creates or updates an MLRepo entity based on the provided manifest.

Create or Update MLRepo

Get a ml repo by id
Args:
    id: Unique identifier of the ml repo to get
    user_info: Authenticated user information

Returns:
    GetMLRepoResponse: The ml repo

Get Ml Repo

Delete a ml repo
Args:
    id: Unique identifier of the ml repo to delete
    user_info: Authenticated user information

Returns:
    EmptyResponse: Empty response indicating successful deletion

Delete Ml Repo

List ml repos
Args:
    filters: Filters for the ml repos
    user_info: Authenticated user information

Returns:
    ListMLReposResponse: List of ml repos

List Ml Repos

Retrieves a list of all latest applications. Supports filtering by application ID, name, type, and other parameters. Pagination is available based on query parameters.

List Applications

Create a new Application Deployment based on the provided manifest.

Create Application Deployment

Get Application associated with the provided application ID.

Get Application

Delete Application associated with the provided application ID.

Delete Application

Fetch all deployments for a given application ID with optional filters such as deployment ID or version. Supports pagination.

List Application Deployments

Get Deployment associated with the provided application ID and deployment ID.

Get Deployment details

Pause a running application by scaling to 0 replicas

Pause an Application

Resume a paused application by scaling back to the original number of replicas

Resume an Application

Cancel an ongoing deployment associated with the provided application ID and deployment ID.

Cancel an Ongoing Deployment

List Job Runs for provided Job Id. Filter the data based on parameters passed in the query

List Job Runs

Get Job Run for provided jobRunName and jobId

Get Job Run

Delete Job Run for provided jobRunName and jobId

Delete Job Run

Trigger Job for provided deploymentId or applicationId

Trigger Job

Terminate Job for provided deploymentId and jobRunName

Terminate Job

Get Model

Delete Model

List Models

Get Model Version

Delete Model Version

List Model Versions

Apply

Delete

Get Artifact

Delete Artifact

List Artifacts

Get Artifact Version

Delete Artifact Version

List Artifact Versions

Get Signed Urls For Artifact Version

Create Multi Part Upload

Stage Artifact Version

List Files For Artifact Version

Mark Stage Failure

Get Prompt

Delete Prompt

List Prompts

Get Prompt Version

Delete Prompt Version

List Prompt Versions

List the secret groups associated with a user along with the associated secrets for each group. Filtered with the options passed in the query fields.

List Secret Groups

Creates a secret group with secrets in it. A secret version for each of the created secret is created with version number as 1. The returned secret group does not have any secrets in the <em>associatedSecrets</em> field. A separate API call should be made to fetch the associated secrets.

Create Secret Group

Get Secret Group associated with provided secretGroup id

Get Secret Group

Updates the secrets in a secret group with new values. A new secret version is created for every secret that has a modified value. Returns the updated secret group. The returned secret group does not have any secrets in the <em>associatedSecrets</em> field. A separate API call should be made to fetch the associated secrets.

Update Secret Group

Deletes the secret group, its associated secrets and secret versions of those secrets.

Delete Secret Group

List Secrets associated with a Secret Group.

List Secrets of Secret Group

List secrets associated with a user filtered with optional parameters passed in the body.

List Secrets

Get Secret

Deletes a secret and its versions along with its values.

Delete Secret

List the active deployments that are associated with a secret.

List Associated Active Deployments

Fetches deployment specifications for a model version or a HuggingFace model URL.Pagination is available based on query parameters.

Get Deployment Specifications

Fetches finetuning specifications for a model version or a HuggingFace model URL.Pagination is available based on query parameters.

Get Finetuning Specifications

List all users of tenant filtered by query and showInvalidUsers. Pagination is available based on query parameters.

List Users

This endpoint allows tenant administrators to update the roles of a user within their tenant.

Update User Roles

Get User associated with provided User id

Get User

Invite User

Resend Invite

Deactivate user associated with the provided email within the tenant.

Deactivate User

Activate user associated with the provided email within the tenant.

Activate User

Retrieve all teams associated with the authenticated user. If the user is a tenant admin, returns all teams for the tenant. Pagination is available based on query parameters

Getting Started

Service

Job

ML Repository

Workflow

Large Language Models (LLMs)

Workbenches: Notebooks and SSH

Volumes

Async Service

Secret Management

Collaboration and Access Control

Deploying On Your Own Cloud

HOW TOs

References

Architecture

GCP Architecture Diagram

Access Policies

Getting Started

Service

Job

ML Repository

Workflow

Large Language Models (LLMs)

Workbenches: Notebooks and SSH

Volumes

Async Service

Secret Management

Collaboration and Access Control

Deploying On Your Own Cloud

HOW TOs

References

​GCP Architecture Diagram

​Access Policies

GCP Architecture Diagram

Access Policies