Deploy instance of a Python class
In this guide, we will learn how to deploy an instance of a Python class
as a Service
using servicefoundry
.
A benefit of deploying a class over plain functions is it allows you to do some initialisations using the __init__
method as well as keep some state between requests if you wish to.
Before we begin,
-
You need to have the
servicefoundry
library installed and log in using theservicefoundry login --host https://app.example.yourdomain.com
command. If you do not have the library installed follow the instructions here. -
Select a workspace on the the workspace page. Copy the workspace FQN. We will use the workspace FQN to deploy the service in that workspace.
Writing the class that we will deploy
File Structure:
.
├── inference.py
└── requirements.txt
Note
- Your class module should be locally importable.
- The inputs and return type of your methods in the class must be type annotated. See https://docs.python.org/3/library/typing.html for additional reference.
- The data type of the inputs and return values of the methods in the class should be json serialisable types.
These include:
None
(SeeOptional
in Python type hints)str
int
bool
float
List
Dict
inference.py
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
class Model:
def __init__(self, model_fqn: str):
self.tokenizer = AutoTokenizer.from_pretrained(model_fqn)
self.model = AutoModelForSeq2SeqLM.from_pretrained(model_fqn)
def infer(self, input_text: str) -> str:
input_ids = self.tokenizer(input_text, return_tensors="pt").input_ids
outputs = self.model.generate(input_ids)
return self.tokenizer.decode(outputs[0], skip_special_tokens=True)
requirements.txt
transformers==4.17.0
torch==1.12.1
Deployment
- Here we will use the
FunctionService
class fromservicefoundry
library to define and deploy the service. - We can use the
register_class
method to deploy the instance of the class. - All the public callable methods of the instance will be deployed as HTTP POST APIs.
Note
The instance properties, attributes, and methods starting with
_
(E.g.def _infer_helper(self, ...)
) will not be registered as routes. - While deploying, we automatically install the requirements defined in the
requirements.txt
file.
File Structure:
.
├── inference.py
├── requirements.txt
└── deploy.py
Note
Before running the command given below, for deploying functions as a service, you need to install all the dependencies locally.
For that run the following command:
pip install -r requirements.txt
deploy.py
import argparse
import logging
from servicefoundry.function_service import FunctionService
from inference import Model
logging.basicConfig(level=logging.INFO)
parser = argparse.ArgumentParser()
parser.add_argument("--workspace_fqn", required=True, type=str)
args = parser.parse_args()
service = FunctionService(
name="t5-small",
resources=Resources(memory_request=1000, memory_limit=1500),
)
service.register_class(
Model,
init_kwargs={"model_fqn": "t5-small"},
name="t5-small"
)
service.deploy(workspace_fqn=args.workspace_fqn)
# NOTE:- You can run the service locally using
# service.run().join()
Notice that we pass the class itself as the first argument not an instance of it. And init_kwargs
are the keyword arguments that would be passed to initialize this class later when Service is deployed.
You can deploy using the following command by passing in your workspace FQN.
python deploy.py --workspace_fqn "<YOUR-WORKSPACE-FQN>"
Important
The above command will only upload the contents of the current directory. Therefore, when you import the function/module for registration, ensure that the import statement is relative to the directory from where you are deploying.
After you deploy the service, you will get an output like below,
INFO:servicefoundry:Method 'infer' from `Model:t5-small` will be deployed on path 'POST /t5-small/infer'.
INFO:servicefoundry:Deploying application 't5-small' to 'v1:local:my-ws-2'
INFO:servicefoundry:Uploading code for service 't5-small'
INFO:servicefoundry:Uploading contents of '/Users/debajyotichatterjee/work/truefoundry-examples/deployment/function/hf_inference_function_deployment'
INFO:servicefoundry:.tfyignore not file found! We recommend you to create .tfyignore file and add file patterns to ignore
INFO:servicefoundry:Deployment started for application 't5-small'. Deployment FQN is 'v1:local:my-ws-2:t5-small:v5'
INFO:servicefoundry:Service 't5-small' will be available at
'https://t5-small-my-ws-2.tfy-ctl-euwe1-devtest.devtest.truefoundry.tech'
after successful deployment
INFO:servicefoundry:You can find the application on the dashboard:- 'https://app.devtest.truefoundry.tech/applications/cl84s4a9z008x1qrubflzflg8?tab=deployments'
You can find the host of the deployed service in the following section in the above logs.
INFO:servicefoundry:Service 't5-small' will be available at
'https://t5-small-my-ws-2.tfy-ctl-euwe1-devtest.devtest.truefoundry.tech'
after successful deployment
Note
Swagger API documentation will be available on the root path.
Click on the host link to open the docs
You can send requests to the deployed service by using the code snippet below. Pass the host using the --host
command line argument.
File Structure:
.
├── inference.py
├── requirements.txt
├── deploy.py
└── send_request.py
send_request.py
import argparse
from urllib.parse import urljoin
import requests
parser = argparse.ArgumentParser()
parser.add_argument("--host", required=True, type=str)
args = parser.parse_args()
response = requests.post(
urljoin(args.host, "/t5-small/infer"),
json={"input_text": "translate English to German: Hello world."},
)
print(response.json())
Updated 19 days ago