module dataset.py

class DataDirectory

property created_at

Get the time at which DataDirectory was created

property created_by

Get the information about who created the DataDirectory

property description

Get description of the DataDirectory

property fqn

Get fqn of the DataDirectory

property metadata

Get metadata for the current DataDirectory

property name

Get the name of the DataDirectory

property storage_root

Get storage_root of the DataDirectory

property updated_at

Get the information about the when the DataDirectory was updated

function add_files

Logs File in the DataDirectory.

Args:

• file_paths (List[truefoundry.ml.DataDirectoryPath], optional): A list of pairs of (source path, destination path) to add files and folders to the DataDirectory contents. The first member of the pair should be a file or directory path and the second member should be the path inside the artifact contents to upload to.

         E.g. >>> data_directory.add_files(
              ...     file_paths=[
                         truefoundry.ml.DataDirectoryPath("foo.txt", "foo/bar/foo.txt"),
                         truefoundry.ml.DataDirectoryPath("tokenizer/", "foo/tokenizer/"),
                         truefoundry.ml.DataDirectoryPath('bar.text'),
                         ('bar.txt', ),
                         ('foo.txt', 'a/foo.txt')
                      ]
              ... )
         would result in
         .
         └── foo/
             ├── bar/
             │   └── foo.txt
             └── tokenizer/
                 └── # contents of tokenizer/ directory will be uploaded here

Examples:

import os
import truefoundry.ml as tfm

with open("artifact.txt", "w") as f:
     f.write("hello-world")

client = tfm.get_client()
data_directory = client.get_data_directory_by_fqn(fqn="<data_directory-fqn>")

data_directory.add_files(
     file_paths=[tfm.DataDirectoryPath('artifact.txt', 'a/b/')]
)

function download

Download an file or directory to a local directory if applicable, and return a local path for it.

Args:

• download_path: Relative source path to the desired files.
• path: Absolute path of the local filesystem destination directory to which to download the specified files. This directory must already exist. If unspecified, the files will either be downloaded to a new uniquely-named directory.
• overwrite: if to overwrite the files at/inside dst_path if they exist

Returns:

• str: Absolute path of the local filesystem location containing the desired files or folder.

Examples:

import truefoundry.ml as tfm

client = tfm.get_client()
data_directory = client.get_data_directory_by_fqn(fqn="<your-artifact-fqn>")
data_directory.download(download_path="<your-desired-download-path>")

classmethod from_fqn

Get the DataDirectory to download contents or load them in memory

Args:

• fqn (str): Fully qualified name of the data directory.

Returns:

• DataDirectory: An DataDirectory instance of the artifact

Examples:

import truefoundry.ml as tfm

client = tfm.get_client()
data_directory = tfm.DataDirectory.from_fqn(fqn="<data_directory-fqn>")

function list_files

List all the files and folders in the data_directory.

Args:

• path: The path of directory in data_directory, from which the files are to be listed.

Returns:

• str: Absolute path of the local filesystem location containing the desired files or folder.

Examples:

import truefoundry.ml as tfm

client = tfm.get_client()
data_directory = client.get_data_directory_by_fqn(fqn="<your-artifact-fqn>")
files = data_directory.list_files()
for file in files:
     print(file.path)

function update

Updates the current instance of the DataDirectory.

Examples:

import truefoundry.ml as tfm

client = tfm.get_client()
data_directory = client.get_data_directory_by_fqn(fqn="<your-data_directory-fqn>")
data_directory.description = 'This is the new description'
data_directory.update()

class DataDirectoryPath

DataDirectoryPath(src, dest)