Documentation
BlogsCreate Account
Start typing to search…

Artifact and Model Version Files Structure Migration Notice

📘

Note

Outlined migration actions are required if you are using one of the following in your code.

Models and Artifacts automatically downloaded using the Service - Artifacts Download feature will keep working without any extra actions.

Existing Structure

Up until truefoundry<0.5.0, when you logged an artifact or model all the contents would be logged internally under a files/ folder for an artifact version and files/model/ for a model version

E.g.

Artifact

from truefoundry.ml import get_client

client = get_client()
client.log_artifact(
    ml_repo="...",
    name="...",
    artifact_paths=[
        ("my-artifact-file-1.txt", None),
        ("my-artifact-file-2.txt", None),
        ("my-folder/", None),
    ]
)

Uploaded Structure:

.
├── .truefoundry/
│   └── metadata.json
└── files/
    ├── my-artifact-file-1.txt
    ├── my-artifact-file-2.txt
    ├── my-folder/
    │   └── ... 
    └── ...

Model

from truefoundry.ml import get_client

client = get_client()
client.log_artifact(
    ml_repo="...",
    name="...",
    model_file_or_folder="./sample-model/",
)

Uploaded Structure:

.
├── .truefoundry/
│   └── metadata.json
└── files/
    └── model/
        ├── model.safetensorts
        ├── config.json
        └── ...

This was to do some additional book keeping on our end.

New Structure

Startingtruefoundry>=0.5.0 (and new UI), we have removed the files/ hierarchy and now all the files will be stored directly under the root of the artifact. That means for the same code above, after logging using the new sdk or UI you will see the following structure:

Artifact

.
├── my-artifact-file-1.txt
├── my-artifact-file-2.txt
└── ...

Model:

.
├── model.safetensorts
├── config.json
└── ...

❗️

Upcoming Automated Migration

The older structure is now considered deprecated. We will perform an automated migration and starting January 2025 we will require alllog_artifact, log_model, ArtifactVersion.download and ModelVersion.download invocations to usetruefoundry>=0.5.0

Please see the following section for required actions.

Required Actions

1. Do not hardcode files/ or files/model/ structure in your code.

👍

You do not need to upgrade truefoundry version for this code change

However, we recommend you upgrade as soon as possible

If you are relying on appending files/ or files/model/ manually to download path

Artifact Version Download

❌ Don't do this

import os
from truefoundry.ml import get_client

client = get_client()
artifact_version = client.get_artifact_version_by_fqn("...")
download_path = "some/path/to/download/"

artifact_version.download(path=download_path)
download_path = os.path.join(download_path, "files")
# equivalent to download_path = download_path + "files/"

# Do something with download_path
# e.g. load_files(download_path)

✅ Instead use the return value of download(). It points to the actual contents

import os
from truefoundry.ml import get_client

client = get_client()
artifact_version = client.get_artifact_version_by_fqn("...")
download_path = "some/path/to/download/"

download_path = artifact_version.download(path=download_path)

# Do something with download_path
# e.g. load_files(download_path)

In Summary

import os
from truefoundry.ml import get_client

client = get_client()
artifact_version = client.get_artifact_version_by_fqn("...")
download_path = "some/path/to/download/"

- artifact_version.download(path=download_path)
- download_path = os.path.join(download_path, "files")
- # equivalent to download_path = download_path + "files/"
+ download_path = artifact_version.download(path=download_path)

# Do something with download_path
# e.g. load_files(download_path)

Model Version Download

❌ Don't do this

import os
from truefoundry.ml import get_client

client = get_client()
model_version = client.get_model_version_by_fqn("...")
download_path = "some/path/to/download/"

model_version.download(path=download_path)
download_path = os.path.join(download_path, "files", "model")
# equivalent to download_path = download_path + "files/model/"

# Do something with download_path
# e.g. load_model(download_path)

✅ Instead use the returned download_info from download(). It has a model_dir attribute that points to the actual contents

import os
from truefoundry.ml import get_client

client = get_client()
model_version = client.get_model_version_by_fqn("...")
download_path = "some/path/to/download/"

download_info = model_version.download(path=download_path)
download_path = download_info.model_dir

# Do something with download_path
# e.g. load_model(download_path)

In Summary

import os
from truefoundry.ml import get_client

client = get_client()
model_version = client.get_model_version_by_fqn("...")
download_path = "some/path/to/download/"

- model_version.download(path=download_path)
- download_path = os.path.join(download_path, "files", "model")
- # equivalent to download_path = download_path + "files/model/"
+ download_info = model_version.download(path=download_path)
+ download_path = download_info.model_dir

# Do something with download_path
# e.g. load_model(download_path)

2. Upgrade to truefoundry>=0.5.0

Once you have ensured you are not hardcoding files/ or files/model/ hierarchy in your code. You can safely upgrade truefoundry as it can handle both older and newer structures. No additional code change will be needed.

Compatibility Matrix

APItruefoundry<0.5.0truefoundry>=0.5.0UI
log_artifactUploads in older nested files/structureUploads in new flat structureUploads in new flat structure
ArtifactVersion.downloadCan only handle older nested files/structureBackwards Compatible, Can handle both structures-
log_modelUploads in older nested files/model/structureUploads in new flat structureUploads in new flat structure
ModelVersion.downloadCan only handle older nested files/model/ structureBackwards Compatible, Can handle both structures-

Updated 8 days ago