Jump to Content
Documentation
API Reference
Changelog
Blogs
Create Account
Documentation
Blogs
Create Account
Documentation
API Reference
Changelog
Deploying LLMs
All
Pages
Start typing to search…
Getting Started
About TrueFoundry
Create and Setup your account
TrueFoundry Key Concepts
Create Your First Deployment
Setup for CLI
Generating TrueFoundry API Keys
Service
Introduction to a Service
Deploy your first Service
Deploy Service from a public Github repository
Interacting with your Service
Configuring your service
Dockerize your code
Define Ports and Domains
Define Resources
Environment Variables and Secrets
Autoscaling
Benchmarking your deployed service using Locust
Liveness/Readiness Probe
Rollout Strategy
Mounting Files
Access Cloud Services like S3
Download Models and Artifacts
Using Fractional GPUs
Using TPUs
Redirect and Mirror Traffic
Patch your Kubernetes Deployment (Advanced)
Update, Rollback, Promote
Monitor your Service
Set up Alerts
Log & Monitor Custom Metrics
Set up CI/CD
Job
Introduction to a Job
Running your first Job
Deploy and Run Job from a public GitHub repository
Interacting with your Job
Monitor your Job
Configuring your Job
Configure Job Trigger
Parameterize a job
Environment Variables and Secrets
Mounting Volumes
Set Retries And Timeout
Set Concurrency Limit
Access data from S3 or other clouds
Add Alerts to your Job
ML Repository
Introduction to ML Repo
Quick Start
Guides
Creating a ML Repo
Creating a run
Log and Get Parameters
Log and Get Metrics
Log and Get Artifacts
Log and Get Models
Log and Get Data
Add Tags
Log Images
Log Plots
Add Artifacts via UI
API Reference
Runs
Artifact
Models
Data Directory
Workflow
Introduction to Workflow
Creating your first workflow
Interacting with workflow
Workflow Concepts
Guides
Creating a workflow with different container images
Using dockerfile for python function task
Using raw container task
Adding environment variable
Configuring Resources
Creating a Conditional Task
Creating a Map Task
Creating a Cron Workflow
Task Config
Attaching Mounts
Passing Files/Artifacts between tasks
Example of Task config with different parameters
Adding retries and handling failures
Infra Set-Up for Workflows
Large Language Models (LLMs)
Introduction to LLMOps
Deploying LLMs
Deploy LLMs using Nvidia TensorRT-LLM (TRT-LLM)
Finetuning LLMs
RAG
LLM Gateway
Intro to LLM Gateway
Adding Models to LLM Gateway
Unified API and Code Generation
Request Logging and Metrics
Configure Rate Limits
Notebooks : VSCode
Launch Jupyter Notebook
Launch an SSH Server
Volumes
Introduction to Volume
Creating and Utilizing Volume
Interacting with your Volume
Creating Statically Provisioned Volumes
Async Service
Introduction to Async Service
Deploy your first Async Service
Configure async service
Configure Ports
Configure Queue
Autoscaling
Queue Integrations
AWS SQS
Monitor your Async Service
Secret Management
Introduction to Secrets
Manage Secrets
Collaboration and Access Control
Collaboration and Access Control
Deploying On Your Own Cloud
Modes of Deployment
Deploy Compute Plane
AWS
Overview
Architecture
Requirements
Setting up DNS and TLS in AWS
GCP
Overview
Architecture
Requirements
Setting up DNS and TLS in GCP
Azure
Overview
Architecture
Requirements
Setting up DNS and TLS in AKS
Deploy Control Plane Only
AWS
GCP
Azure
Installation with Helm
Dev Mode installation
Generic
Advanced Configuration
LoadBalancer
Add Certificate for TLS
Metrics
Logging
Cost Monitoring
Notebook Controller
TFY Agent
Support self signed CA certificates
Enabling Git Integrations
Adding pre and post build scripts
LLM Gateway
Integrations
Integration Provider - AWS
Integration Provider - GCP
Integration Provider - Azure
Integration Provider - Git
Other Integrations
SSO Integration
OpenID Connect with Azure AD
OpenID Connect with Okta
SAML v2 with Azure AD
Deploy Truefoundry in an Air-gapped Environment
HOW TOs
Migrate Pytorch Sagemaker Endpoints to TrueFoundry Platform
Create custom K8s objects
Model Deployment
Scikit Learn
XGBoost
References
Pre-built Truefoundry Machine Learning Container Images
Guide to inference model on truefoundry
FAQs & Glossary
FAQs
Infrastructure and Setup FAQs
Security and Compliance FAQs
Access Control FAQs
Hosted Notebooks and VSCode FAQs
Model Deployment FAQs
Model Training and LifeCycle FAQs
LLM Deployment And Finetuning FAQs
LLM Gateway FAQs
Application Templates FAQs
Cost Optimisation FAQs
Deploying LLMs
Updated 8 days ago
What’s Next
Send Requests to the Deployed LLM
