Runs

A run represents a single experiment which in the context of Machine Learning is one specific model (say Logistic Regression), with a fixed set of hyper-parameters. Metrics, and parameters (details below) are all logged under a specific run.

Creating Runs in Truefoundry

A run is an entity that represents a single experiment. Create a run at the beginning of your script or notebook to start capturing system metrics.
Note : User also needs to create the ml_repo (if it doesn't exist) either from the dashboard or with create_ml_repo function as illustrated below.

from truefoundry.ml import get_client

client = get_client()
client.create_ml_repo("iris-demo")
run = client.create_run(ml_repo="iris-demo", run_name="svm-model")
# Your code here.
run.end()

You can organize multiple runs under a single ml_repo. For example, the run svm-model will be created under the ml_repo iris-sklearn-example.

You can view these runs in the TrueFoundry dashboard.

Accessing Runs in TrueFoundry

To interact with runs in Truefoundry, you can use the provided methods in the TrueFoundryClient class. Here are the different possibilities to access runs:

Get a Run by ID

To retrieve an existing run by its ID, use the get_run_by_id method:

client = TrueFoundryClient()  
run = client.get_run_by_id("run_id_here")

Get a Run by Fully Qualified Name (FQN)

If you have the fully qualified name (FQN) of a run, which follows the pattern tenant_name/ml_repo/run_name, you can use the get_run_by_fqn method:

client = TrueFoundryClient()  
run = client.get_run_by_fqn("tenant_name/ml_repo/run_name")

Get All Runs for a Project

To retrieve all the runs' names and IDs for a project, use the get_all_runs method:

client = TrueFoundryClient()  
runs_df = client.get_all_runs(ml_repo="project_name_here")

Search Runs

You can search for runs that match specific criteria using the search_runs method:

client = TrueFoundryClient()  
runs = client.search_runs(  
    ml_repo="project_name_here",  
    filter_string="metrics.accuracy > 0.75",  
    order_by=["metric.accuracy DESC"],  
)
for run in runs:  
    print(run)

FAQs

What is a ml_repo?

A ml_repo embodies the high-level goal of the experiments, like "predicting the sentiment of product reviews". To reach the goal, you can experiment with different machine learning algorithms with different parameters. A single run represents a single experiment. TrueFoundry helps you organize these runs and find the best-performing ones under a ml_repo.

How can I create a ml_repo?

A ml_repo is automatically created when you call the create_run method. A ml_repo is identified by it's owner and name.

Can anyone create a run under my ml_repo?

No. TrueFoundry provides ml_repo-level authorization. If someone in your team wants to view or create a run under your ml_repo, you need to add them as a collaborator to your ml_repo.

How can I create a run under a ml_repo owned by someone else?

You can pass the ml_repo argument in the the create run. You should at least have WRITE permission for the ml_repo. If you don't have write access to the ml_repo, the admin needs to provide you atleast WRITE permission to the ml_repo.

client.create_ml_repo("iris-demo")

run = client.create_run(
    ml_repo="iris-demo",
    run_name="svm-model",
)
# Your code here.
run.end()

Can I use runs as a context manager?

Yes, we can use runs as a context manager. A run will be automatically ended after the execution exits the with block.

client.create_ml_repo("iris-demo")

run = client.create_run(ml_repo="iris-demo", run_name="svm-model")
with run:
    # Your code here.
    ...

# No need to call run.end()

Are run names unique?

Yes. run names under a ml_repo are unique. If a run name already exists, we add a suffix to make it unique.
If you do not pass a run name while creating a run, we generate a random name.

from truefoundry.ml import get_client

client = get_client()
client.create_ml_repo("iris-demo")
run = client.create_run(ml_repo="iris-demo")

print(run.run_name)
run.end()

How runs are identified?

Runs are identified by by their id.

from truefoundry.ml import get_client

client = get_client()
client.create_ml_repo("iris-demo")
run = client.create_run(ml_repo="iris-demo")

print(run.run_id)
run.end()