Deploy instance of a Python class

In this guide, we will learn how to deploy an instance of a Python class as a Service using servicefoundry.

A benefit of deploying a class over plain functions is it allows you to do some initialisations using the __init__ method as well as keep some state between requests if you wish to.

Before we begin,

1. You need to have the servicefoundry library installed and log in using the servicefoundry login --host https://app.example.yourdomain.com command. If you do not have the library installed follow the instructions here.

2. Select a workspace on the the workspace page. Copy the workspace FQN. We will use the workspace FQN to deploy the service in that workspace.

Writing the class that we will deploy

File Structure:

.
├── inference.py
└── requirements.txt

📘

Note

1. Your class module should be locally importable.
2. The inputs and return type of your methods in the class must be type annotated. See https://docs.python.org/3/library/typing.html for additional reference.
3. The data type of the inputs and return values of the methods in the class should be json serialisable types.

These include:

• None (See Optional in Python type hints)
• str
• int
• bool
• float
• List
• Dict

inference.py

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM


class Model:
    def __init__(self, model_fqn: str):
        self.tokenizer = AutoTokenizer.from_pretrained(model_fqn)
        self.model = AutoModelForSeq2SeqLM.from_pretrained(model_fqn)

    def infer(self, input_text: str) -> str:
        input_ids = self.tokenizer(input_text, return_tensors="pt").input_ids
        outputs = self.model.generate(input_ids)
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

requirements.txt

transformers==4.17.0
torch==1.12.1

Deployment

• Here we will use the FunctionService class from servicefoundry library to define and deploy the service.
• We can use the register_class method to deploy the instance of the class.
• All the public callable methods of the instance will be deployed as HTTP POST APIs.

  📘

  Note

  The instance properties, attributes, and methods starting with _ (E.g. def _infer_helper(self, ...)) will not be registered as routes.

• While deploying, we automatically install the requirements defined in the requirements.txt file.

File Structure:

.
├── inference.py
├── requirements.txt
└── deploy.py

📘

Note

Before running the command given below, for deploying functions as a service, you need to install all the dependencies locally.

For that run the following command: pip install -r requirements.txt

deploy.py

import argparse
import logging

from servicefoundry.function_service import FunctionService

from inference import Model

logging.basicConfig(level=logging.INFO)

parser = argparse.ArgumentParser()
parser.add_argument("--workspace_fqn", required=True, type=str)
args = parser.parse_args()

service = FunctionService(
    name="t5-small",
    resources=Resources(memory_request=1000, memory_limit=1500),
)

service.register_class(
  Model, 
  init_kwargs={"model_fqn": "t5-small"}, 
  name="t5-small"
)

service.deploy(workspace_fqn=args.workspace_fqn)
# NOTE:- You can run the service locally using
# service.run().join()

Notice that we pass the class itself as the first argument not an instance of it. And init_kwargs are the keyword arguments that would be passed to initialize this class later when Service is deployed.

You can deploy using the following command by passing in your workspace FQN.

python deploy.py --workspace_fqn "<YOUR-WORKSPACE-FQN>"

🚧

Important

The above command will only upload the contents of the current directory. Therefore, when you import the function/module for registration, ensure that the import statement is relative to the directory from where you are deploying.

After you deploy the service, you will get an output like below,

INFO:servicefoundry:Method 'infer' from `Model:t5-small` will be deployed on path 'POST /t5-small/infer'.
INFO:servicefoundry:Deploying application 't5-small' to 'v1:local:my-ws-2'
INFO:servicefoundry:Uploading code for service 't5-small'
INFO:servicefoundry:Uploading contents of '/Users/debajyotichatterjee/work/truefoundry-examples/deployment/function/hf_inference_function_deployment'
INFO:servicefoundry:.tfyignore not file found! We recommend you to create .tfyignore file and add file patterns to ignore
INFO:servicefoundry:Deployment started for application 't5-small'. Deployment FQN is 'v1:local:my-ws-2:t5-small:v5'
INFO:servicefoundry:Service 't5-small' will be available at
'https://t5-small-my-ws-2.tfy-ctl-euwe1-devtest.devtest.truefoundry.tech'
after successful deployment
INFO:servicefoundry:You can find the application on the dashboard:- 'https://app.devtest.truefoundry.tech/applications/cl84s4a9z008x1qrubflzflg8?tab=deployments'

You can find the host of the deployed service in the following section in the above logs.

INFO:servicefoundry:Service 't5-small' will be available at
'https://t5-small-my-ws-2.tfy-ctl-euwe1-devtest.devtest.truefoundry.tech'
after successful deployment

📘

Note

Swagger API documentation will be available on the root path.

Click on the host link to open the docs

You can send requests to the deployed service by using the code snippet below. Pass the host using the --host command line argument.

File Structure:

.
├── inference.py
├── requirements.txt
├── deploy.py
└── send_request.py

send_request.py

import argparse
from urllib.parse import urljoin

import requests

parser = argparse.ArgumentParser()
parser.add_argument("--host", required=True, type=str)
args = parser.parse_args()

response = requests.post(
    urljoin(args.host, "/t5-small/infer"),
    json={"input_text": "translate English to German: Hello world."},
)
print(response.json())