TensorFlow Serve (TFServe)

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data.

In this example, we will deploy a simple MNIST model using TensorFlow Serving. You can find the code for this example here.

The key files are:

• train.py: Trains the model and exports it in SavedModel format.
• batching.config: Contains dynamic batching configuration.
• requirements.txt: Contains the dependencies.

Usually the SavedModel format checkpoint is self contained and can be deployed without writing any extra code.

​

Exporting the model in SavedModel format

...
model.fit(train_images, train_labels, epochs=epochs)
version = 1
export_path = os.path.join("./models", "mnist", str(version))
print("export_path = {}\n".format(export_path))
model.export(export_path)
print("\nSaved model to", export_path)

​

Running the server locally

1. Install the dependencies

pip install -r requirements.txt

2. Run the server

export MODEL_DIR="$(pwd)/models/mnist"
tensorflow_model_server --model_name=mnist --model_base_path=$MODEL_DIR --enable_batching --batching_parameters_file=./batching.config --rest_api_port=8000 --rest_api_timeout_in_ms=10000 --enable_model_warmup

3. Test the server

curl -X POST -H "Content-Type: application/json" --data @./example.json http://0.0.0.0:8000/v1/models/mnist/versions/1:predict

The output should look like this:

{
  "predictions": [
    [
      0.999824345,
      5.03408815e-10,
      2.40962e-05,
      5.19650811e-08,
      1.13485561e-10,
      5.45465309e-06,
      3.39081362e-06,
      1.45393031e-09,
      0.000138094867,
      4.5494553e-06
    ]
  ]
}

​

Deploying the model with TrueFoundry

To deploy the model, we need to package both the model file and the code. To do this, we can follow the steps below:

from truefoundry.ml import get_client, TensorflowFramework
client = get_client()

model_version = client.log_model(
    ml_repo="demo-models",
    name="mnist-tensorflow",
    model_file_or_folder="./models/mnist/",
    description="MNIST model in Tensorflow saved format",
    framework=TensorflowFramework()
)
print("Model version FQN:", model_version.fqn)

.
└── model/
    └── mnist/  # Log this directory containing the model version
        └── 1/

tensorflow_model_server --model_name=mnist --model_base_path=$(MODEL_DIR) --enable_batching --batching_parameters_file=/batching.config --rest_api_port=8000 --rest_api_timeout_in_ms=10000 --enable_model_warmup