Autoscaling

1. If one replica goes down because of any reason, the entire service goes down. Having 2-3 replicas at minimum helps provide more fault-tolerance.
2. A single replica might not be able to take all of the incoming traffic and hence cause client side errors.

​

Setting Fixed Number of Replicas

​

Enable Autoscaling

1. CPU Usage: If the application’s cpu usage goes up when traffic increases, this can be a good parameter to autoscale on. For e.g., let’s say your application (one replica) consumes 0.3 vCPU on steady state - however, as traffic goes up, the CPU usage starts increasing to a max of 1 CPU. In this case, setting autoscaling to trigger when CPU usage is greater than 0.6 can be a good idea.
2. Requests Per Second (RPS): This is the easiest to reason about and calculate. We can benchmark our service to decide how many requests per second can be served by one replica using benchmarking tools like Locust. Let’s say one replica can serve 10 requests per second without an degradation in quality (increase in latency or errors). If we expect the traffic to vary from 50 requests per second to 200 requests per second, we can set the minimum replicas to 5, maximum to 20 and set the rps autoscaling metric to 10.
3. Time Based Autoscaling: This can be useful if the traffic patterns shift based on timezones, or you want to shut down dev workloads during non-office hours. For e.g, if you want to scale up replicas only between Monday to Friday 9AM to 9PM, you can set time based scheduling with cron start schedule as 0 9 * * 1-5 and end schedule as 0 21 * * 1-5.

​

Configure Autoscaling for an Application

1. Min Replicas: The minimum number of replicas to scale down to.
2. Max Replicas: The maximum number of replicas to scale up to.
3. Autoscaling Metric: The metric to autoscale on.
4. Autoscaling Value: The value of the metric to autoscale on.
5. Polling Interval (Advanced): This defines the time interval at which the metrics are checked to decide if the service needs to be scaled up/down. The default values is 30 seconds. You can make it lower to 10 seconds in case you want to scale more quickly.

​

How to decide the autoscaling metric value

• Determining the service’s capacity in terms of handling requests per second.
• Assessing the threshold of concurrent requests, indicating the number of users the service can accommodate simultaneously.
• Analyzing how latency fluctuates with increasing traffic volume.

​

How to benchmark a service

1

Install Locust

Copy

Ask AI

pip install locust

2

Write the Locust File

We need to write a simple class which implements a function to send requests to your deployed service. You will need to replace

locust_benchmark.py

Copy

Ask AI

from locust import FastHttpUser, task

class HelloWorldUser(FastHttpUser):
    @task
    def hello_world(self):
        self.client.post(
            # No need to mention host here
            "/path/to/your/endpoint",  
            # Add your request payload here
            json={
                "model": "<Add your deployed service name>",
                "prompt": [
                    "This is a test prompt"
                ],
                "max_tokens":80,
                "ignore_eos":True
            }
        )

In the code above, we are sending a POST request to the deployed service with the payload mentioned above.

If your service is deployed at https://service-name.example.com and you want to benchmark the API at https://service-name>.example.com/path/to/your/endpoint, you only need to mention /path/to/your/endpoint in the code above.

You can read this doc to understand more about writing a locust file.

3

Run the Locust File locally

Copy

Ask AI

locust -f locust_benchmark.py

Once you run this, you can open http://localhost:8089 in a browser window. You will find the UI like this:

In the section of host, paste the endpoint of your service: https://service-name.example.com/ in the example above.

4

Send requests to the service to benchmark

Once you paste the link, you can click on “Start Swarming” after setting the following parameters:

• Number of Users: Number of concurrent users that will bombard your service
• Spawn Rate: If multiple users are selected, the rate at which it will create new users (this can be 1 by default)

Once you start swarming, you can see the results on the dashboard:

Once this is setup, you can edit and increase the number of users from top by clicking on Edit. You can view the detailed charts by clicking on “Charts” tab as shown in the picture. The results look something like this.

5

Deploy the locust service on Truefoundry

While you can benchmark your service by running Locust locally, its usually recommended to deploy the locust service on the same cluster in which your service is deployed. This helps in reducing the latency from the locust code to your service and helps provide more accurate results.

The latencies depicted in Locust while running on your local machine might not be accurate since its dependent on your internet speed. Moving it to the cluster helps provide a more stable and fast network.

You can deploy your locust code as a service on truefoundry. To deploy, place the following 2 files in a directory (Let’s call it locust_service).

• locust_benchmark.py(shown above)
• requirements.txt

locust_benchmark.py

requirements.txt

Copy

Ask AI

from locust import FastHttpUser, task

class HelloWorldUser(FastHttpUser):
    @task
    def hello_world(self):
        self.client.post(
            # No need to mention host here
            "/path/to/your/endpoint",  
            # Add your request payload here
            json={
                "model": "<Add your deployed service name>",
                "prompt": [
                    "This is a test prompt"
                ],
                "max_tokens":80,
                "ignore_eos":True
            }
        )

Once you have this ready, please go to TrueFoundry UI and follow the following steps:

1. Click on New Deployment button on top right of your screen.
2. Select your workspace
3. Select Code from Laptop
4. Click on Next
5. Follow the guide from the UI
6. In the deployment form, name the service as locust-service, choose Python Code (I don't have Dockerfile) option and set the command as locust -f locust_benchmark.py.

Run locust with only 1 replica. Running it with more than 1 replica will not be useful since only 1 pod can run benchmarking and it will cause wierd issues in the Locust UI.

Once the deployment of the locust service completes. you can open the endpoint of the locust service and see the same UI that we saw above while running it locally.

6

Benchmark your service to get the autoscaling metric value

Once the deployment of the locust service completes, you can open the endpoint of the locust service and see the same UI that we saw above while running it locally. To benchmark your service (let’s call it my-service), follow the guidelines below:

1. Run 1 replica of service-a. Its important to benchmark with 1 replica to understand how many requests per second can be served by 1 replica.
2. Start with a few users (1-10) and gradually keep increasing it.
3. We should increase the number of users till either the latency starts increasing beyond our point of tolerance or we start seeing errors from the service.
4. Observe the CPU and memory usage of the service at this time from the metrics on Truefoundry portal.

5. The CPU usage should be going up as we increase the number of users and hence requests per second.

In most cases, if CPU usage is not going up with requests per second going up, or the service is incurring high latencies or errors even when CPU usage is not close to requested value - it denotes an inefficiency in the code or configuration.

6. We can try increasing the cpu requested if we want per pod to allow it to handle more requests per second. Let’s say we can handle 10 requests per second with 1 CPU. Going more than 10 requests per seconds results in higher latencies. In this case we can set the autoscaling metric to requests per second and the value to somewhere around 7-8 to have some buffer in each replica.
7. Set the autoscaling criteria and redeploy the service. Now if you increase the number of users in Locust, you should be able to see the pods autoscale to multiple replicas and requests per second goes above 10.