You can provide link to any HuggingFace model and Truefoundry will be automatically able to deploy the model as an API. We analyze the HuggingFace model tags and files to understand which is the deployment framework
to serve the model and automatically generate the deployment manifests for the model.
Some models might not be supported or might need additioal inputs since the HuggingFace tags might be missing or incorrect.
In some cases, you might need a HuggingFace token to access the model. This is needed if you are trying to download your own private models or using gated models
like Llama which require you to accept the license and terms of use.
In this case, you will need to create a secret with your HuggingFace token. You can create a secret following the guide here.
You can then enable the Is it a private model? toggle and enter the secret name in the dialog.
Here’s the list of the most commonly used task types that are supported in Truefoundry and how to infer from the model endpoint.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
These are commonly known as LLMs. We use vLLM, SGLang or TRT-LLM to deploy these models. To read more on the deployment of LLMs, please refer to the LLM Deployment guide.
