Model Deployment
Note
While using TrueFoundry python SDK
typeis not a required field in any of the imported classes
ModelDeployment
Description
Describes the configuration for the model deployment
Schema
{
"type": "string",
"name": "string",
"model_source": {},
"env": null,
"endpoint": {
"host": "string",
"path": "string"
},
"grpc": false,
"resources": {
"cpu_request": 0.2,
"cpu_limit": 0.5,
"memory_request": 200,
"memory_limit": 500,
"ephemeral_storage_request": 1000,
"ephemeral_storage_limit": 2000,
"gpu_count": 0,
"shared_memory_size": 64,
"node": {}
},
"replicas": 1,
"mounts": [
{}
],
"labels": {
"property1": "string",
"property2": "string"
}
}
Properties
| Name | Type | Required | Description |
|---|---|---|---|
| type | string | true | +value=model-deployment |
| name | string | true | Name of the model deployment. This uniquely identifies this deployment in the workspace. > Name can only contain alphanumeric characters and '-' and can be atmost 20 characters long |
| model_source | object | true | Specify the model to be deployed |
| env | object¦null | false | Configure environment variables to be injected in the service. Docs |
| endpoint | Endpoint | false | Describes an endpoint configuration |
| grpc | boolean | true | Use grpc |
| resources | Resources | false | Describes the resource constraints for the application so that it can be deployed accordingly on the cluster To learn more you can go here |
| replicas | integer | true | Replicas of service you want to run |
| mounts | [object] | false | Configure data to be mounted to model pod(s) |
| labels | object | false | Add labels to service metadata |
Endpoint
Description
Describes an endpoint configuration
Schema
{
"host": "string",
"path": "string"
}
Properties
| Name | Type | Required | Description |
|---|---|---|---|
| host | string | true | Host e.g. ai.example.com, app.truefoundry.com |
| path | string | false | Path e.g. /v1/api/ml/, /v2/docs/ |
Model Source
The modules below help define the model source for the ModelServer
TruefoundryModelRegistry
Description
Truefoundry model registry (deploy a model from TFY model registry)
Schema
{
"type": "string",
"model_version_fqn": "string"
}
Properties
| Name | Type | Required | Description |
|---|---|---|---|
| type | string | true | +value=tfy-model-registry |
| model_version_fqn | string | true | The FQN of the model version that you want to deploy. |
HuggingfaceModelHub
Description
Huggingface model hub (deploy a model from HF model hub)
Schema
{
"type": "string",
"model_id": "string",
"pipeline": "string",
"model_library": "transformers"
}
Properties
| Name | Type | Required | Description |
|---|---|---|---|
| type | string | true | +value=hf-model-hub |
| model_id | string | true | The name of the model that you want to deploy. Example:- t5-small, philschmid/bart-large-cnn-samsum. |
| pipeline | string | false | Pipeline to use for inference. If not set, we will try to infer the task name for the given model.Examples: summarization, text-generation, text-classification, etc.You can read more about HF pipelines here:- https://huggingface.co/docs/transformers/pipeline_tutorial You can find a list of tasks here:- https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/pipelines#transformers.pipeline.task Note: With model_library= sentence-transformers only feature-extraction pipeline is supported |
| model_library | string | true | Which library to use to load the model. Should be either transformers or sentence-transformers |
Enumerate Values
| Property | Value |
|---|---|
| model_library | transformers |
| model_library | sentence-transformers |
Updated 23 days ago