Supported APIs
Moderation API
Supported Providers
- OpenAI
Use the moderations endpoint to identify content that may violate usage policies, such as text or images involving violence, hate speech, harassment, self-harm, or sexual content.
You can use two models for this endpoint:
omni-moderation-latest
: This model and all snapshots support more categorization options and multi-modal inputs.text-moderation-latest
(Legacy): Older model that supports only text inputs and fewer input categorizations. The newer omni-moderation models will be the best choice for new applications.
You can add these models in your OpenAI provider account.
Code Snippets
Here’s a full example output, where the input is an image from a single frame of a war movie. The model correctly predicts indicators of violence in the image, with a violence
category score of greater than 0.8.
Key Fields
flagged
:true
if any content is considered harmful.categories
: Dictionary of category flags,true
means the input falls under that harmful category.category_scores
: Probability scores (0 to 1) for each category.category_applied_input_types
: Indicates whether the category was triggered by the text or image.