Request Logging and Metrics

Gateway provides you with the feature to log requests/response and metrics like tokens consumed with each request. Here's how you can navigate the logs:

Enable Prompt Logging from the Playground page.
Requests - Go to the 'Requests' tab to see all the requests triggered by each user across the models. You can filter these requests using model name, username, etc.
Clicking on each prompt will let you view the prompt, response, input/output tokens and cost
Metrics - Navigate to the Metrics Page for a visual and tabular representation of various metrics such as request/user/sec, latency, etc

Filter Requests

You can use one or a combination of filters on the Requests page, such as model_name, username, and output tokens, to refine your search results and find specific data that meet your criteria.

Programatically access request/response pairs

Clicking on each prompt on the Requests page allows you to view the exact prompt used, the model's response, the number of input and output tokens, and the associated cost.

Visual and Tabular Metrics

You can filter by model and user for a specified period on the Metrics Page to view various metrics, including input tokens, output tokens, total requests, and tokens cost.This can be useful in model and user activity analysis as well as cost management.

These metrics are accessible in both graphical and tabular formats for easy analysis and interpretation.

Visual representation

Tabular representation