SSL and DNS setup in Azure

How to setup DNS and terminate the traffic via TLS.

In this section we will see how we can setup DNS using Azure DNS and terminate the ingress traffic using SSL from the certificate generated by cert-manager and Let's encrypt.

If you are using a different DNS provider then almost same changes will reflect in the DNS provider and cert-manager settings. To perform all the below tasks cert-manager and tfy-istio-ingress must be installed already.

Setting up DNS

DNs settings can be done in two ways

Public load balancer

If you want your endpoints to be exposed to the outer world so that they can be accessed from the internet then the load balancer type must be public. For this no change needs to be made in the tfy-istio-ingress. By default the application will create a public load balancer with public endpoint.

To get the IP of this public endpoint

kubectl get svc -n istio-system tfy-istio-ingress -ojsonpath='{.status.loadBalancer.ingress[0].ip}'

Once this endpoint is available make sure to point your domain name with A record to the above public IP in your DNS provider.

Private load balancer

There are use cases when endpoints are to be kept private. For this we have to use a private load balancer in Azure. To get a private IP for load balancer

  • Go to Deployments page in the left panel.

  • Select your application name (tfy-istio-ingress) from the Helm section and click on Edit from the three dots in the right side.

  • Edit the section with the below gateway.annotations

    gateway:
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    tfyGateway:
      name: tfy-wildcard
      spec:
        servers:
          - tls:
              httpsRedirect: true
            port:
              name: http-tfy-wildcard
              number: 80
              protocol: HTTP
            hosts:
              - "*.infra.truefoundry.com"
          - port:
              name: https-tfy-wildcard
              number: 443
              protocol: HTTP
            hosts:
              - "*.infra.truefoundry.com"
        selector:
          istio: tfy-istio-ingress
    
    
  • Once done you can see your load balancer will have a private IP address

    $ k get svc -n istio-system tfy-istio-ingress -ojsonpath='{.status.loadBalancer.ingress[0].ip}'
    10.224.0.8
    
  • Point this private load balancer IP to the domain name in your favourite DNS provider. If you want to use Azure DNS and you don't have a DNS zone created follow the below document.

Setting up hosted Zone in Azure DNS

  • Authenticate to Azure cli

  • Create the DNS hosted zone

    $ az network dns zone create \
    --name demo2.truefoundry.com \
    --resource-group tfy-datascience \
    --query nameServers
    
  • The above command will output list of the name servers that must be pointed in your main DNS resolver as NS record. This is done so that any traffic coming to demo2.truefoundry.com (in this case) will come to the hosted zone in Azure DNS.

  • To check if you name servers are pointed correctly, run the below command

    dig -t NS +short demo2.truefoundry.com
    
  • Now we will create a record with the load balancer's IP so that request coming to *.demo2.truefoundry.com goes to the load balancer's IP

    export LOAD_BALANCERIP="<LOADBALANCER IP HERE>"
    
    az network dns record-set a add-record \                                   
    --ipv4-address $LOAD_BALANCERIP \
    --record-set-name "*" \  
    --resource-group tfy-datascience \
    --zone-name demo2.truefoundry.com
    
  • To test the record

    dig -t A something.demo2.truefoundry.com +short
    

SSL/TLS using cert-manager

Once we have our DNS set up we need to set up the cert-manager to issue certificates. In this section we will use AzureDNS as the DNS provider to authenticate for DNS challenges that are created by Issuers in cert-manager . You can check Cert-manager DNS provider configuration to configure DNS challenges for your preferred DNS provider. Below documentation gives a brief overview on how to use Azure DNS to create DNS challenges and issue certificates.

  • If you don't have workload identity feature enabled for your AKS cluster use the command to enable it. If you have used Azure AKS page to create AKS cluster then it is enabled by default.

    az aks update \
        --name ${CLUSTER} \
        --resource-group ${RESOURCE_GROUP} \
        --enable-oidc-issuer \
        --enable-workload-identity # ℹ️ This option is currently only available when using the aks-preview extension.
    
  • Make sure to create a managed identity. If you have used Azure AKS page to create AKS then we have already created a managed identity under the name tfy-user-identity.

    • To create a new managed identity use the below command and save the output Client ID
      export IDENTITY_NAME=cert-manager
      PRINCIPAL_ID=$(az identity create --name "${IDENTITY_NAME}" --query principalId)
      
    • If you have already created the managed identity or want to use an already existing one then run the following command
      # As per the installations steps of AKS cluster
      PRINCIPAL_ID=$(az identity show \
      --name tfy-user-identity \
      --resource-group tfy-datascience \
      --query principalId -otsv)
      
  • Get the ID of the DNS zone

    DNS_ZONE_ID=$(az network dns zone show \
    --name demo2.truefoundry.com \
    --resource-group tfy-datascience \
    --query id -otsv)
    
  • Assign the managed identity DNS contributor role to the DNS zone

    az role assignment create \
    --assignee $PRINCIPAL_ID \
    --role "DNS Zone Contributor" \
    --scope $DNS_ZONE_ID
    
  • Assigning the service account in kubernetes to use the managed identity. This is called federated identity where we are assigning the managed identity so that is can be used by

    • Set these variables with proper values

      # service account of cert-manager
      SERVICE_ACCOUNT_NAME=cert-manager
      
      # namespace of cert-manager
      SERVICE_ACCOUNT_NAMESPACE=cert-manager
      
      # get the resource group
      # make sure to use your resource group if the value is different
      RESOURCE_GROUP=tfy-datascience
      
      # get the cluster name
      # make sure to use the right cluster name if the values is different
      CLUSTER_NAME=tfy-aks-cluster
      
      # if you are using identity created in azure aks installation page
      IDENTITY_NAME=tfy-user-identity
      
      # get the OIDC_ISSUER_URL
      OIDC_ISSUER_URL=$(az aks show \
      --resource-group $RESOURCE_GROUP \
      --name $CLUSTER_NAME \
      --query "oidcIssuerProfile.issuerUrl" -o tsv)
      
    • Creating federated identity

      az identity federated-credential create \
        --name "cert-manager" \
        --identity-name "${IDENTITY_NAME}" \
        --issuer "${OIDC_ISSUER_URL}" \
        --resource-group "${RESOURCE_GROUP}" \
        --subject "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
      
  • We will edit the cert-manager application that is installed in the cluster.

    • Go to Deployments section from the left panel and edit you application cert-manager from the Helm tab
    • Pass these values in the values section and click submit
      installCRDs: true
      extraArgs:
        - --issuer-ambient-credentials
      podLabels:
        azure.workload.identity/use: "true"
      serviceAccount:
        labels:
          azure.workload.identity/use: "true"
      
    • --issuer-ambient-credentials is a flag in cert-manager to use credentials from metadata
    • The labels passed above will help cert-manager to authenticate to Azure.
  • Create Issuer . Make sure to create it in istio-system namespace

    • Fill in these variables

      # your mail ID
      MAIL_ID=<>
      
      # dns zone name
      DNS_ZONE_NAME="demo2.truefoundry.com"
      
      # subscription ID
      AZURE_SUBSCRIPTION_ID=<>
      
      # get managed identity client ID
      IDENTITY_CLIENT_ID=$(az identity show \
      --name $IDENTITY_NAME \
      --resource-group $RESOURCE_GROUP \
      --query 'clientId' -otsv)
      
    • kubectl apply -f - <<EOF
      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: demo2-issuer
        namespace: istio-system
      spec:
        acme:
          # give your email ID which will get notificate when the certificate gets expired
          email: $MAIL_ID
           # staging server is used for testing, it will create certificate
           # which are not publicly accepted. For valid production certificates
           # use https://acme-v02.api.letsencrypt.org/directory as acme.server
          server: https://acme-staging-v02.api.letsencrypt.org/directory
          privateKeySecretRef:
            name: demo2-privkey
          solvers:
          - dns01:
              azureDNS:
                hostedZoneName: $DNS_ZONE_NAME
                resourceGroupName: $RESOURCE_GROUP
                subscriptionID: $AZURE_SUBSCRIPTION_ID
                environment: AzurePublicCloud
                managedIdentity:
                  clientID: $IDENTITY_CLIENT_ID
      EOF
      
  • Create a certificate

    • Make sure to point the issuer name from the above code correctly in issuerRef.name
    • Create the below certificate
      kubectl apply -f - <<EOF
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: demo2-cert
        namespace: istio-system
      spec:
        # the secret in which the cert will be pasted once provisioned
        # it MUST BE PRESENT IN THE istio-system NAMESPACE
        secretName: demo2-tls
        issuerRef:
          # Issuer name from the previous step
          name: demo2-issuer
        dnsNames:
        - "demo2.truefoundry.com"
        - "*.demo2.truefoundry.com"
      EOF
      
    • It will take sometime for certificate to get created
      $ kubectl get cert -n istio-system
      NAME         READY   SECRET      AGE
      demo2-cert   True    demo2-tls   4m29
      
    • The secret demo2-tls will contain the certificates
      $ kubectl get secret -n istio-system -l controller.cert-manager.io/fao=true
      NAME        TYPE                DATA   AGE
      demo2-tls   kubernetes.io/tls   2      5m47s
      

Adding the secret in ingress

  • Now the secret is created we need to add this secret to tfy-istio-ingress application for it terminate to SSL.

  • Go to Deployments section from the left panel and edit application tfy-istio-ingress .

  • Enter the secret in the values section at tfyGateway.spec.servers[1].tls.credentialName and click Submit

    tfyGateway:
      name: tfy-wildcard
      spec:
        servers:
          - tls:
              httpsRedirect: true
            port:
              name: http-tfy-wildcard
              number: 80
              protocol: HTTP
            hosts:
              - "*.demo2.truefoundry.com"
              - "demo2.truefoundry.com"
          - port:
              name: https-tfy-wildcard
              number: 443
              protocol: HTTPS
            hosts:
              - "*.demo2.truefoundry.com"
              - "demo2.truefoundry.com"
            tls:
              mode: SIMPLE
              credentialName: demo2-tls
        selector:
          istio: tfy-istio-ingress
    
    
  • Once the secret is added we need to enable the base domain URL from the cluster settings.

    • Go to Integrations tab and click on Edit section of the cluster.
    • Add the base domain URL
  • Now these domain URLs can be used to expose your endpoints in the deployments section.

    • Go to deployments section and create a new service (let is be nginx)
    • Expose it at port 80.
    • Create the service and access the URL. The format will be
      https://nginx-<workspace-name>-80.demo2.truefoundry.com/