Define Ports and Domains

When you create a Service, you need to specify the ports that the Service will listen on. You can decide to expose a port in which case you will have to attach a domain to the service.

Not exposing the port

If you do not expose a service, you get a URL of the form servicename-workspacename.svc.cluster.local:port which can only be called from services within the cluster.

Exposing the port

This will enable the service to be called from anywhere using the endpoint - other users or services living outside your cluster will also be able to access it. In this case, you will need to map a domain URL to your service. TrueFoundry supports both host-based routing and path-based routing.

If you use host-based routing, two services s1 and s2 will have URLs of the form s1.yourdomain.com and s2.yourdomain.com.

In the case of path-based routing, the URLs will be of the form yourdomain.com/s1 and yourdomain.com/s2. If you decide to enable path-based routing, please ensure the application that you have deployed supports path-based routing.

Domains are configured in the cluster. You can view which domains are accessible to you from here.

Host-based routing will be preferred especially if you are deploying some UI-based applications.

Configuring the Port for your Service

Via the User Interface (UI)

Via the Python SDK

The Service class has a ports argument where you can pass a list of Port objects. Each Port object represents a port that the service will listen on. In the Port field you can specify the Port Number, host for your Endpoint and Path (Optional)

📘

Picking a value for host

Providing a host value depends on the base domain urls configured in the cluster settings, you can learn how to find the base domain urls available to you here

For e.g. If your base domain url is *.truefoundry.your-org.com then a valid value can be fastapi-your-workspace-8000.truefoundry.your-org.com.

Alternatively if you have a non wildcard based domain url e.g. truefoundry.your-org.com, then a valid value can be truefoundry.your-org.com/fastapi-your-workspace-8000

import logging

from truefoundry.deploy import Build, Service, DockerFileBuild, Port

logging.basicConfig(level=logging.INFO)
service = Service(
    name="your-service",
    image=Build(build_spec=DockerFileBuild()),
+   ports=[
+       Port(
+           port=8080,
+           # Uses subdomain based routing
+           host="fastapi-your-workspace-8080.your-organization.com",
+           # Alternatively we can use path based routing - E.g.
+           # host="your-site.your-organization.com",
+           path="/fastapi/v1/"
+       )
+   ]
)
service.deploy(workspace_fqn="YOUR_WORKSPACE_FQN")

What if you don't know which port to use?

Usually, most of the commonly used applications will come with a way to define the port. We have listed it for some of the most commonly used frameworks in ML.

FastAPI (Uvicorn) - Default port is 8000. This can be overridden using the command uvicorn main:app --port 8001

Gradio App - Launch your Gradio app as follows and enter the port number 8080.

gradio.app.launch(server_name="0.0.0.0", port=8080)

Now you need to specify the same port number in the service configuration. In this case 8080

Streamlit - Default port is 8501.

Adding Basic authentication to Service

Ideally when deploying a Service it is a good idea to protect publicly exposed ports using some form of authentication. You can add a username password based authentication to your endpoints or JWT-based authentication to your services.

Via the User Interface (UI)

Via the Python SDK

We can add basic (username and password) authentication to a Port of the Service in the Python definition using truefoundry.deploy.BasicAuthCreds )

from truefoundry.deploy import Build, Service, Port, BasicAuthCreds, DockerFileBuild

service = Service(
    name="your-service",
    image=Build(build_spec=DockerFileBuild()),
    ports=[
        Port(
            port=8080,
            host="fastapi-my-workspace-8080.truefoundry.my-org.com",
+           auth=BasicAuthCreds(
+                username="hello",
+                password="pass"
+           )
        )
    ]
)
service.deploy(workspace_fqn="YOUR_WORKSPACE_FQN")

Sending request to your Authenticated Endpoint

Upon deploying the service, the endpoint will now display a padlock 🔒 icon, indicating that authentication is required for access.

Browser Access

When attempting to access the endpoint from a web browser, you will be prompted to enter your credentials.

Python Access

When using Python, you can pass the credentials in the following manner:

import requests

endpoint = 'https://fastapi-my-workspace-8080.truefoundry.my-org.com/'

response = requests.get(endpoint, auth=('hello', 'pass'))

Curl Access

When using cURL, you'll need to include the credentials as part of the command:

curl -v -X GET --user hello:pass https://fastapi-my-workspace-8080.truefoundry.my-org.com/

Adding JWT authentication to Service

You can configure JWT-based authentication for your services hosted on TrueFoundry. By leveraging JWT (JSON Web Tokens), you can ensure that only authenticated traffic can access your endpoints.

The token is then sent to your service as part of the HTTP requests. Your service verifies the JWT using a public key from the IDP's JSON Web Key Set (JWKS). This process ensures that the JWT is valid and has not been tampered with, allowing your service to authenticate the request based on the token's claims.

Integrating Third-Party Identity Providers

Third-party identity providers like Amazon Cognito or Azure AD can be used to manage user authentication and issue JWTs. These providers handle user sign-in processes and generate JWTs that assert the user's identity and attributes. You need to configure your service to trust and validate JWTs issued by these IDPs, typically by specifying the IDP’s “issuer” URL and JWKS URI in your authentication settings.

Adding JWT Authentication to your service

To add JWT Authentication to your service, you can enable Custom Security Rule and then fill out the issuer and the JWKS URI. Additionally, you can verify claims in the token by providing a list of accepted values per key.

Sending request to your Authenticated Endpoint

While sending request to the authenticated, you need to provide the token in the Authorizationheader. This header carries the token, allowing your service to verify and authenticate the request based on the credentials and claims present in the JWT.

Here is an example of how to send a request using curl, a common command-line tool used for making HTTP requests:

curl -X GET "https://your-service-endpoint.truefoundry.cloud/api/data" \
     -H "Authorization: Bearer YOUR_JWT_HERE"

Unauthenticated requests would return 403.

Specifying Host

When we deploy services, we often want to put them behind a well-defined recognizable URL instead of load balancer URLs or IP addresses. E.g.https://app.your-organization.com or https://your-site.your-organization.com/app/v1/. There are two steps involved in this:

1. If your organization owns domains, integrate them before proceeding. For instructions, please refer to the following guide
2. Construct endpoint URLs on top of your domains.

Identifying Available Domains

Before specifying the host for your service URL, determine which domains are available to you. This helps in choosing the appropriate endpoint structure.

You will see that there are two possible domain types:

• Wildcard Domain: *.your-organization.com
• Standard Domain: your-organization.com

The approach to constructing the URL will differ based on what type of domain we are using.

Constructing the URL

a) Wildcard Domain:

For wildcard domains, specify the unique identifier before the .your-organization.com part. For example, your-site-name.your-organization.com.

b) Standard Domain:

For standard domains, specify the unique identifier after the your-organization.com part. For example, your-organization.com/your-site-name.