Skip to main content
If you are deploying models as APIs, you will need the model file to be present along with the service so that you can load and serve it. There are a few ways people usually end up doing this: Bake the model file into the docker image This works, however, it has the following disadvantages:
  1. The docker image size can become huge if the model file is big.
  2. We will need to rebuild the docker image if the model file changes.
  3. Every time the pod needs to come up on a new machine, it will end up downloading the model file again.
Download the model file from S3 at the start of the server
  1. In this case, every time a pod comes up, we need to download the model first which leads to high startup time.
  2. It also means more networking costs every time the models are downloaded.
Create a volume, download the model file on the volume, and mount the volume to your service This approach has a lot of benefits:
  1. Smaller docker image size leads to faster startup
  2. Faster loading of models from volume leads the faster startup times
  3. Cost savings because of not having to download the model repeatedly.
However, this can be a bit cumbersome to set up since the steps need to be done manually. Also, one more thing to be careful of here is the upgrade process from one model version to another. The key issues here are:
  1. Upgrading requires manual intervention of having to download the new version of the model on the volume.
  2. If we swap the older model on the volume with the newer model, and a pod restarts just after that, it will lead to one pod using the newer version of the model, even though we haven’t started the deployment process.
To overcome the above issues, you can use the Artifacts Download feature in TrueFoundry which automatically provisions a volume, downloads the model onto the volume, mounts the volume and also handles upgrades safely. The way this works during the first deployment is:
As you can see in the diagram above, the user will need to define the link to the model to be downloaded and an environment variable in which TrueFoundry will provide the path where the model is present. The reason this is done this way is to provide a smooth and failsafe upgrade process. When you are deploying again, the following flow will be followed:
In this scenario, the model is not downloaded if there is no change in the artifact version. However, if there is a change, TrueFoundry downloads the new model at a different path and then restarts all the servers with the new path in the environment variable. It also cleans up the older model once the rollout is fully complete. This makes it easy to switch back to the older version during a rollout if anything goes wrong.

​
Using Artifact Download Feature to cache models

​
Using the Cached Model

In your Service code, to use the cached model, you can utilize the Download Path Environment Variable you specified while configuring the Download and Cache Artifact. You can do one of these two things, Get it directly in your code using os.environ.get() or Pass it as a command-line argument if your service needs it that way.

  1. ​
    Read the download path from environment variable

For example, if you specify MODEL_ID_1 as theDownload Path Environment Variable, to download model.joblib file, you can access the cached model like this: 
import os
import joblib

# Get the model_path using the environment variable
model_path_1 = os.environ.get("MODEL_ID_1")

# Load the model using the specified path
model_1 = joblib.load(os.path.join(model_path_1, "model.joblib"))

  1. ​
    Pass the download path as a command-line argument

For example, in case you are deploying an LLM on top of the vLLM model server using the vllm/vllm-openai:v0.2.6 docker image. The following takes the model and tokenizer as a command-line argument. You can pass the Download Path Environment Variable in the arguments.
⌘I