Setting up DNS and TLS in GCP

To host any service/model endpoints a domain has to be used to expose them to the external world or to an internal network. Below document will help you to set the same in your GCP GKE standard cluster. Any number of domains can be setup for your cluster.

Setting up DNS

There are two kind of domains that you can setup for TrueFoundry workloads

  1. Wild card domains - *.example.com, *.tfy.example.com, *.ml.example.com
  2. Non wild card domains - tfy.example.com, dev.example.com, prod.example.com

Wild card domains (recommended)

In wild card domains a subdomain wildcard is dedicatedly used to resolve endpoints in the GKE cluster. Some of the samples are given below where example.com is your domain. The services will be exposed like

  • service1.tfy.example.com
  • service2.tfy.example.com

Non wild card domains

In non-wild card domains a dedicated domain is used to resolve endpoints. Some of the samples for service endpoints will look like

  • tfy.example.com/service1
  • tfy.example.com/service2

Load balancer IP address

Once a domain name is decided a DNS record is to be mapped with the load balancer IP address in the GKE cluster. To get the load balancer's IP address run the following command

kubectl get svc -n istio-system tfy-istio-ingress -ojsonpath={.status.loadBalancer.ingress[0].ip}

Create a DNS record in your cloud DNS or your DNS provider with the following details

Record TypeRecord NameRecord value
A*.tfy.example.comLOADBALANCER_IP_ADDRESS

Setting up TLS

We support creation of TLS certificates with the help of cert-manager. cert-manager in our case use LetsEncrypt which issues the certificate. Cert-manager can connect with various DNS provider so that LetsEncrypt can verify if you are the true owner of the domain or not. Below example is to create TLS certificate using GCP cloud DNS. Feel free to use other DNS provider by following the cert-manager documentation.

πŸ“˜

Workload identity

For onboarding an existing cluster, ensure that workload identity is enabled for the GKE cluster. For clusters created with the help of ocli, it is enabled by default.

1. Create an IAM role and attach it to a GCP serviceaccount

export PROJECT_ID=""
export DNS_ROLE_NAME=""
export GCP_SERVICEACCOUNT_NAME="<>"

gcloud config set project $PROJECT_ID
gcloud iam service-accounts create $GCP_SERVICEACCOUNT_NAME --display-name "$GCP_SERVICEACCOUNT_NAME"

# Assign the IAM serviceaccount permissions to access DNS records
gcloud iam roles create custom_dns_role \
    --project=$PROJECT_ID \
    --title=$DNS_ROLE_NAME \
    --permissions=dns.resourceRecordSets.create,dns.resourceRecordSets.delete,dns.resourceRecordSets.get,dns.resourceRecordSets.list,dns.resourceRecordSets.update,dns.changes.create,dns.changes.get,dns.changes.list,dns.managedZones.list

gcloud projects add-iam-policy-binding $PROJECT_ID \
--condition=None \
--member=serviceAccount:$GCP_SERVICEACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com \
--role="projects/$PROJECT_ID/roles/custom_dns_role"

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT_ID.svc.id.goog[cert-manager/cert-manager]" \
    $GCP_SERVICEACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com

2. Created a hosted zone

To make GCP you DNS provider, create a hosted zone. It is not a compulsory requirement and you can use a domain from an already existing hosted DNS zone in GCP.

gcloud dns managed-zones create example-com \
    --description="DNS zone for example.com" \
    --dns-name=example.com \
    --visibility=public --format=json | jq '.[].nameServers'

The above command will print the nameservers which have to setup in your main DNS provider (for e.g. GoDaddy or NameCheap) as an NS record.

3. Install cert-manager

  1. Head over to the Integrations tab in the left panel and select the right cluster in which you want to install cert-manager. Click on the Three dots at the left-panel and select Manage Applications.
    1. Install cert-manager, if not already installed by creating a workspace.
  2. If cert-manager is already installed, go to Deployments -> Helm -> Filter the charts with the cluster name and click on edit from the three dots at the right.
  3. In the values section, ensure the below lines exists. Make sure to replace the $GCP_SERVICEACCOUNT_NAME and $PROJECT_ID with the serviceaccount name and the project ID
    extraArgs:
      - --issuer-ambient-credentials
    serviceAccount:
      create: true
      annotations:
        iam.gke.io/gcp-service-account: $GCP_SERVICEACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com
    

4. Creating an issuer and a certificate

  1. Download the kubeconfig file for your cluster
    gcloud container clusters get-credentials CLUSTER_NAME  --zone ZONE  --project PROJECT
    
  2. Create the following issuer by replacing the example text below to your preferred name. Please replace the PROJECT_ID as well.
    kubectl apply -f - <<EOF
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: example-com-issuer
      namespace: istio-system
    spec:
      acme:
        email: [email protected]
        server: https://acme-v02.api.letsencrypt.org/directory
        privateKeySecretRef:
          name: example-com-privkey
        solvers:
        - dns01:
            cloudDNS:
              project: $PROJECT_ID
    EOF
    
  3. Create the certificate by referencing the above issuer example-com-issuer in the certificate with the replaced issuer name. Also add the dnsNames accordingly
    kubectl apply -f - <<EOF
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: example-com-cert
      namespace: istio-system
    spec:
      secretName: example-com-tls
      duration: 2160h # 90d
      renewBefore: 360h # 15d
      issuerRef:
        # Issuer name from the previous step
        name: example-com-issuer
      dnsNames:
      - "example.com"
      - "*.example.com"
    EOF
    
  4. Check the status of the certificate by running the following command. Wait for the certificates to go in the Ready state.
    kubectl get certificates -n istio-system
    

🚧

Certificate is not ready

If the certificate is not in ready state for more then 10 minutes, it means that there is some issue in the access or the domain name. Please check the logs of cert-manager pods in the cert-manager namespace for more details.

5. Attaching the TLS secret to the Load balancer

  1. Once the certificate is created a secret will be present in the istio-system namespace with the name given in .spec.secretName while creating a certificate object.
  2. Head over to the Deployments -> Helm -> filter helm chart for your cluster -> tfy-istio-ingress and then add the secret in .tfyGateway.spec.servers for HTTPS you need to add the tls.mode and tls.credentialName. Please ensure port.protocol is HTTPS for port 443
    tfyGateway:
      name: tfy-wildcard
      spec:
        selector:
          istio: tfy-istio-ingress
        servers:
          - hosts:
              - "*..example.com"
              - "example.com"
            port:
              name: http-tfy-wildcard
              number: 80
              protocol: HTTP
            tls:
              httpsRedirect: true
          - hosts:
              - "*.example.com"
              - "example.com"
            port:
              name: https-tfy-wildcard
              number: 443
              protocol: HTTPS # make sure to keep it HTTPS
            tls:
              mode: SIMPLE
              credentialName: example-com-tls
    

6. Adding the domain in the cluster metadata

  1. Head over to the Integrations section in the platform and click on the Edit in the cluster.
  2. Enable Show advanced fields from the bottom and add enable the Base Domain URL section.
  3. Add the domain URL.