Introduction to service

Services

Truefoundry Service is a continuously running application that typically provides a set of APIs for interaction.

Use Cases

Services are ideal when requests can arrive at any time and prompt responses are crucial. Some typical use cases for services include:

  1. Hosting Realtime Model Inference (e.g., Flask, FastAPI)
  2. Powering Dynamic Backend for Websites
  3. Building Model Demos (e.g., Streamlit, Gradio)

Key Considerations when building a service

A Service typically exposes APIs on designated ports. These ports can be mapped to domains, enabling external calls to the Service. We need to be mindful of the following things while deploying a service:

  1. Dockerizing the code that will be deployed
  2. Defining the ports on which the service can be exposed
  3. [Optional] Mapping domains to the port using which other applications can talk to our service.
  4. [Optional] Defining environment variables and secrets to be injected into the code
  5. [Optional] Defining an autoscaling policy in case of production workloads
  6. [Best Practice] Defining the Liveness / Readiness Probe
  7. [Best Practice] Define a rollout strategy
  8. [Optional] Mount files / volumes to your service
  9. Set up Monitoring Dashboards
  10. Set up alerting