Getting Started

In this guide, we’ll deploy a FastAPI service for solving the Iris classification problem. The problem involves predicting the species of an iris flower based on its sepal length, sepal width, petal length, and petal width. There are three species: Iris setosa, Iris versicolor, and Iris virginica.

We’ve already created a FastAPI service for the Iris classification problem, and you can find the code in our GitHub Repository.

Please visit the repository to familiarize yourself with the code you’ll be deploying. The project files are organized as follows:

Directory Structure

.
├── app.py - Contains FastAPI code for inference.
├── iris_classifier.joblib - The model file.
└── requirements.txt - Lists dependencies.

Getting Started With Deployment

To deploy a service, you’ll need a workspace. If you don’t have one, you can create it using this guide: Creating a Workspace or seek assistance from your cluster administrator in case you don’t have permission to create a workspace.

In Truefoundry, you can either deploy code from your Github repository or from your local machine in case the code is not pushed to a Github repository.

Use these configs for the deployment form:

Repo URL: https://github.com/truefoundry/getting-started-examples
Path to build context: ./deploy-model-with-fastapi/
Command: uvicorn app:app —host 0.0.0.0 —port 8000
Port: 8000

What we did above:

In the example above, we only had Python code and a requirements.txt. We didn’t have a prewritten docker file - so we chose the Python Code option - to let TrueFoundry templatize a Dockerfile from the details provided about the application and build the Docker image for us.

We give these details in the Build context field, where we specify the directory in the GitHub repository where our service code resides (./deploy-model-with-fastapi/). We also specify the command that we need to use to run our service (uvicorn app:app --host 0.0.0.0 --port 8000).

Finally, we specify the port that we want our service to listen on (8000).

Use these configs for the deployment form:

Repo URL: https://github.com/truefoundry/getting-started-examples
Path to build context: ./deploy-model-with-fastapi/
Command: uvicorn app:app —host 0.0.0.0 —port 8000
Port: 8000

What we did above:

Finally, we specify the port that we want our service to listen on (8000).

To deploy from your local machine, you can follow the steps on the UI to get the deployment script.

Once you reach the last step, you will be able to download a deploy.py script that contains the configuration for the deployment. You should place the deploy.py script in the root of your project.

The deploy.py should be placed in the root of your project.

The deploy.py has a field called build_context_path. The build_context_path is considered relative to the location of the deploy.py file.

All the code in the build_context_path will be packaged into a docker image and deployed.

The directory structure will then appear as follows:

.
├── iris_classifier.joblib
├── app.py
├── deploy.py
└── requirements.txt

To deploy, execute the command:

python deploy.py

You should be in the same directory as the deploy.py file when you run the command.

Explanation of the deploy.py script

`deploy.py`

Picking a value for `host`

Providing a host value depends on the base domain urls configured in the cluster settings, you can learn how to find the base domain urls available to you here

For e.g. If your base domain url is *.truefoundry.your-org.com then a valid value can be fastapi-your-workspace-8000.truefoundry.your-org.com.

Alternatively if you have a non wildcard based domain url e.g. truefoundry.your-org.com, then a valid value can be truefoundry.your-org.com/fastapi-your-workspace-8000

import logging
from truefoundry.deploy import Build, PythonBuild, Service, Resources, Port

# Set up logging to display informational messages
logging.basicConfig(level=logging.INFO)

# Create a TrueFoundry **Service** object to configure your service
service = Service(
  	# Specify the name of the service
    name="your-service",
    # Define how to build your code into a Docker image
  	image=Build(
        # `PythonBuild` helps specify the details of your Python Code.
      	# These details will be used to templatize a DockerFile to build your Docker Image
        build_spec=PythonBuild(
          	# Define the command to run the application
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
   		      # Specify the path to requirements file
            requirements_path="requirements.txt",
        )
    ),
  	# Set the ports your server will listen on
    ports=[
        Port(
            port=8000,
          	# Define the host for the service, to learn how to set this view the above callout
            host="your-service-your-workspace-8000.your-organization.ai",
        )
    ],
  	# Define the resource constraints.
    #
    # Requests are the minimum amount of resources that a container needs to run.
    # Limits are the maximum amount of resources that a container can use.
    #
    # If a container tries to use more resources than its limits, it will be throttled or killed.
    resources=Resources(
      # CPU is specified as a number. 1 CPU unit is equivalent to 1 physical CPU core, or 1 virtual core.
      cpu_request=0.2,
      cpu_limit=0.5,

      # Memory is defined as an integer and the unit is Megabytes.
      memory_request=200,
      memory_limit=500,

      # Ephemeral storage is defined as an integer and the unit is Megabytes.
      ephemeral_storage_request=1000,
      ephemeral_storage_limit=2000,
    ),
  	# Define environment variables that your Service will have access to
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

# Deploy the service to the specified workspace, copy workspace FQN using the following guide
# https://docs.truefoundry.com/docs/key-concepts#creating-a-workspace
service.deploy(workspace_fqn="your-workspace-fqn")

To understand the code, you can click the following recipe:

Deploy FastAPI service via python

Setup the project

First, you need to import the following modules:

argparse from Python’s standard library.
logging from Python’s standard library.
Import the necessary classes (Build, PythonBuild, Service, Resources, and Port) from servicefoundry.

Set up logging to display servicefoundry logs by configuring the log level to INFO.

import argparse
import logging
from servicefoundry import Build, PythonBuild, Service, Resources, Port

logging.basicConfig(level=logging.INFO)

Setup Service

Now, it’s time to define the properties of the Service:

Specify the name of the service, which will be its identifier in TrueFoundry’s deployments dashboard.
Define the image with instructions on how to build the container image.
Set up the ports the server should listen to.
Configure the resources for the application.
Define the env variables that should be available in the server environment.

service = Service(
    name="fastapi",
    image=Build(
        build_spec=PythonBuild(
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
            requirements_path="requirements.txt",
        )
    ),
    ports=[
        Port(
            port=8000,
            host=args.host,
        )
    ],
    resources=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=200,
        memory_limit=500,
      	ephemeral_storage_request=1000,
      	ephemeral_storage_limit=2000
    ),
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

Define the Docker Image Build Instructions

For defining how to build your code into a Docker image, use the Build class:

Specify the build_source to determine the source code location. If not provided, the current working directory is used.
Define the build_spec using the PythonBuild class to set up a Python environment.

service = Service(
    name="fastapi",
    image=Build(
        build_spec=PythonBuild(
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
            requirements_path="requirements.txt",
        )
    ),
    ports=[
        Port(
            port=8000,
            host=args.host,
        )
    ],
    resources=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=200,
        memory_limit=500,
      	ephemeral_storage_request=1000,
      	ephemeral_storage_limit=2000
    ),
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

Configure the Python Build

In the PythonBuild class, provide the following arguments:

command: The command to start your service.
requirements_path: The path to your dependencies file.

service = Service(
    name="fastapi",
    image=Build(
        build_spec=PythonBuild(
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
            requirements_path="requirements.txt",
        )
    ),
    ports=[
        Port(
            port=8000,
            host=args.host,
        )
    ],
    resources=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=200,
        memory_limit=500,
      	ephemeral_storage_request=1000,
      	ephemeral_storage_limit=2000
    ),
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

Specify Port

Set the port your server will listen on. In this case, it’s 8000, matching the port in your uvicorn command.

service = Service(
    name="fastapi",
    image=Build(
        build_spec=PythonBuild(
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
            requirements_path="requirements.txt",
        )
    ),
    ports=[
        Port(
            port=8000,
            host=args.host,
        )
    ],
    resources=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=200,
        memory_limit=500,
      	ephemeral_storage_request=1000,
      	ephemeral_storage_limit=2000
    ),
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

Specify resource constraints.

You can specify resource constraints using the Resources class.

cpu_request: Specifies the minimum CPU reserved for the application (0.5 represents 50% of CPU resources).
cpu_limit: Defines the upper limit on CPU usage, beyond which the application is throttled.
memory_request: Specifies the minimum required memory (e.g., 1 means 1 MB).
memory_limit: Sets the maximum memory allowed; exceeding this limit triggers an Out of Memory (OOM) error.
ephemeral storage_request: Specifies the minimum required temporary storage.
ephemeral storage_limit: Sets the maximum temporary storage capacity allowed.

These resource constraints help you control the allocation and utilization of CPU, memory, and temporary storage for your deployments. Temporary storage, often referred to as ephemeral storage, is a short-term storage space for your application’s temporary data.

service = Service(
    name="fastapi",
    image=Build(
        build_spec=PythonBuild(
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
            requirements_path="requirements.txt",
        )
    ),
    ports=[
        Port(
            port=8000,
            host=args.host,
        )
    ],
    resources=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=200,
        memory_limit=500,
      	ephemeral_storage_request=1000,
      	ephemeral_storage_limit=2000
    ),
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

Specifying environment variables

You can also provide environment variables using a dictionary of the format {"env_var_name": "env_var_value". This is helpful for configurations like environment type (dev/prod) or model registry links.

service = Service(
    name="fastapi",
    image=Build(
        build_spec=PythonBuild(
            command="uvicorn app:app --port 8000 --host 0.0.0.0",
            requirements_path="requirements.txt",
        )
    ),
    ports=[
        Port(
            port=8000,
            host=args.host,
        )
    ],
    resources=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=200,
        memory_limit=500,
      	ephemeral_storage_request=1000,
      	ephemeral_storage_limit=2000
    ),
    env={
        "UVICORN_WEB_CONCURRENCY": "1",
        "ENVIRONMENT": "dev"
    }
)

Deploy the service

Use service.deploy() to initiate the deployment. Provide the workspace_fqn (fully qualified workspace name) to specify where the service should be deployed.

service.deploy(workspace_fqn=args.workspace_fqn)

To deploy using Python SDK use:

python deploy.py

Run the above command from the same directory containing the app.py and requirements.txt files.

Exclude files when building and deploying your source code:

To exclude specific files from being built and deployed, create a .tfyignore file in the directory containing your deployment script (deploy.py). The .tfyignore file follows the same rules as the .gitignore file.

If your repository already has a .gitignore file, you don’t need to create a .tfyignore file. Service Foundry will automatically detect the files to ignore.

Place the .tfyignore file in the project’s root directory, alongside deploy.py.

After running the command mentioned above, wait for the deployment process to complete. Monitor the status until it shows DEPLOY_SUCCESS:, indicating a successful deployment.

Once deployed, you’ll receive a dashboard access link in the output, typically mentioned as You can find the application on the dashboard:. Click this link to access the deployment dashboard.

View your deployed service

Congratulations! You’ve successfully deployed your FastAPI service.

Once you click Submit, your deployment will be successful in a few seconds, and your service will be displayed as active (green), indicating that it’s up and running.

You can view all the information about your service following the steps below:

Copy Endpoint URL

To make a request to the Service, you will need the Endpoint URL.

The endpoint URL is the same one you provided while deploying the service in the Ports Section. You can also copy it from the Service UI.

The endpoint will be an internal cluster URL or an external URL based on whether you chose Expose in the Ports configuration. The two cases are as follows:

The service port is exposed and mapped to a domain

In this case, this becomes a public endpoint and you can call this service from anywhere - your own laptop or code running anywhere.

You can add username-password-based authentication to the API in case you don’t want everyone to be able to call this API. For this, you can refer to the following section Add Authentication to Endpoint.

The service port is not exposed

If you have not exposed the port, the endpoint is internal to the cluster and can only be called by other services running in the same cluster (including Jupyter Notebooks running in the same cluster). No one externally can access this service and this is the recommended mode for APIs that don’t need to be exposed for external usage. These APIs will be of the format servicename-workspacename.svc.cluster.local:port

Sending Requests to your Service

Once you deploy the service, you will want to interact with the API in your code or manually using curl or Postman or Python code as shown below:

import json  
from urllib.parse import urljoin

import requests

# Replace this with the value of your endpoint URL

ENDPOINT_URL = "\<YOUR_ENDPOINT_URL>"  # e.g., https://your-service-endpoint.com

response = requests.post(  
    urljoin(ENDPOINT_URL, 'predict'),  
    params={  
        "sepal_length": 7.0,  
        "sepal_width": 3.2,  
        "petal_length": 4.7,  
        "petal_width": 1.4,  
    }  
)

result = response.json()  
print("Predicted Classes:", result["prediction"])

Train and Deploy Models

Service Deployment

Job Deployment

Workflow Deployment

Async Service Deployment

Volumes

ML Repository

Secret Management

Platform

Deploying On Your Own Cloud

HOW TOs

Getting Started

Getting Started With Deployment

What we did above:

What we did above:

Explanation of the deploy.py script

`deploy.py`

Picking a value for `host`

Exclude files when building and deploying your source code:

View your deployed service

Copy Endpoint URL

The service port is exposed and mapped to a domain

The service port is not exposed

Sending Requests to your Service

Getting Started

Train and Deploy Models

Service Deployment

Job Deployment

Workflow Deployment

Async Service Deployment

Volumes

ML Repository

Secret Management

Platform

Deploying On Your Own Cloud

HOW TOs

​Getting Started With Deployment

​What we did above:

​What we did above:

​Explanation of the deploy.py script

​deploy.py

​Picking a value for host

​Exclude files when building and deploying your source code:

​View your deployed service

​Copy Endpoint URL

​The service port is exposed and mapped to a domain

​The service port is not exposed

​Sending Requests to your Service

Getting Started With Deployment

What we did above:

What we did above:

Explanation of the deploy.py script

`deploy.py`

Picking a value for `host`

Exclude files when building and deploying your source code:

View your deployed service

Copy Endpoint URL

The service port is exposed and mapped to a domain

The service port is not exposed

Sending Requests to your Service