SSL and DNS setup in Azure

In this section we will see how we can setup DNS using Azure DNS and terminate the ingress traffic using SSL from the certificate generated by cert-manager and Let's encrypt.

If you are using a different DNS provider then almost same changes will reflect in the DNS provider and cert-manager settings. To perform all the below tasks cert-manager and tfy-istio-ingress must be installed already.

Setting up DNS

DNs settings can be done in two ways

Public load balancer

If you want your endpoints to be exposed to the outer world so that they can be accessed from the internet then the load balancer type must be public. For this no change needs to be made in the tfy-istio-ingress. By default the application will create a public load balancer with public endpoint.

To get the IP of this public endpoint

kubectl get svc -n istio-system tfy-istio-ingress -ojsonpath='{.status.loadBalancer.ingress[0].ip}'

Once this endpoint is available make sure to point your domain name with A record to the above public IP in your DNS provider.

Private load balancer

There are use cases when endpoints are to be kept private. For this we have to use a private load balancer in Azure. To get a private IP for load balancer

Go to Deployments page in the left panel.
Select your application name (tfy-istio-ingress) from the Helm section and click on Edit from the three dots in the right side.

Edit the section with the below gateway.annotations

gateway:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
tfyGateway:
  name: tfy-wildcard
  spec:
    servers:
      - tls:
          httpsRedirect: true
        port:
          name: http-tfy-wildcard
          number: 80
          protocol: HTTP
        hosts:
          - "*.infra.truefoundry.com"
      - port:
          name: https-tfy-wildcard
          number: 443
          protocol: HTTP
        hosts:
          - "*.infra.truefoundry.com"
    selector:
      istio: tfy-istio-ingress

Once done you can see your load balancer will have a private IP address

$ k get svc -n istio-system tfy-istio-ingress -ojsonpath='{.status.loadBalancer.ingress[0].ip}'
10.224.0.8

Point this private load balancer IP to the domain name in your favourite DNS provider. If you want to use Azure DNS and you don't have a DNS zone created follow the below document.

Setting up hosted Zone in Azure DNS

Authenticate to Azure cli

Create the DNS hosted zone

$ az network dns zone create \
--name demo2.truefoundry.com \
--resource-group tfy-datascience \
--query nameServers

The above command will output list of the name servers that must be pointed in your main DNS resolver as NS record. This is done so that any traffic coming to demo2.truefoundry.com (in this case) will come to the hosted zone in Azure DNS.
To check if you name servers are pointed correctly, run the below command
```
dig -t NS +short demo2.truefoundry.com
```

Now we will create a record with the load balancer's IP so that request coming to *.demo2.truefoundry.com goes to the load balancer's IP

export LOAD_BALANCERIP="<LOADBALANCER IP HERE>"

az network dns record-set a add-record \                                   
--ipv4-address $LOAD_BALANCERIP \
--record-set-name "*" \  
--resource-group tfy-datascience \
--zone-name demo2.truefoundry.com

To test the record

dig -t A something.demo2.truefoundry.com +short

SSL/TLS using cert-manager

Once we have our DNS set up we need to set up the cert-manager to issue certificates. In this section we will use AzureDNS as the DNS provider to authenticate for DNS challenges that are created by Issuers in cert-manager . You can check Cert-manager DNS provider configuration to configure DNS challenges for your preferred DNS provider. Below documentation gives a brief overview on how to use Azure DNS to create DNS challenges and issue certificates.

If you don't have workload identity feature enabled for your AKS cluster use the command to enable it. If you have used Azure AKS page to create AKS cluster then it is enabled by default.

az aks update \
    --name ${CLUSTER} \
    --resource-group ${RESOURCE_GROUP} \
    --enable-oidc-issuer \
    --enable-workload-identity # ℹ️ This option is currently only available when using the aks-preview extension.

Make sure to create a managed identity. If you have used Azure AKS page to create AKS then we have already created a managed identity under the name tfy-user-identity.
- To create a new managed identity use the below command and save the output Client ID
```
export IDENTITY_NAME=cert-manager
PRINCIPAL_ID=$(az identity create --name "${IDENTITY_NAME}" --query principalId)
```
- If you have already created the managed identity or want to use an already existing one then run the following command
```
# As per the installations steps of AKS cluster
PRINCIPAL_ID=$(az identity show \
--name tfy-user-identity \
--resource-group tfy-datascience \
--query principalId -otsv)
```

Get the ID of the DNS zone

DNS_ZONE_ID=$(az network dns zone show \
--name demo2.truefoundry.com \
--resource-group tfy-datascience \
--query id -otsv)

Assign the managed identity DNS contributor role to the DNS zone

az role assignment create \
--assignee $PRINCIPAL_ID \
--role "DNS Zone Contributor" \
--scope $DNS_ZONE_ID

Assigning the service account in kubernetes to use the managed identity. This is called federated identity where we are assigning the managed identity so that is can be used by

Set these variables with proper values

# service account of cert-manager
SERVICE_ACCOUNT_NAME=cert-manager

# namespace of cert-manager
SERVICE_ACCOUNT_NAMESPACE=cert-manager

# get the resource group
# make sure to use your resource group if the value is different
RESOURCE_GROUP=tfy-datascience

# get the cluster name
# make sure to use the right cluster name if the values is different
CLUSTER_NAME=tfy-aks-cluster

# if you are using identity created in azure aks installation page
IDENTITY_NAME=tfy-user-identity

# get the OIDC_ISSUER_URL
OIDC_ISSUER_URL=$(az aks show \
--resource-group $RESOURCE_GROUP \
--name $CLUSTER_NAME \
--query "oidcIssuerProfile.issuerUrl" -o tsv)

Creating federated identity

az identity federated-credential create \
  --name "cert-manager" \
  --identity-name "${IDENTITY_NAME}" \
  --issuer "${OIDC_ISSUER_URL}" \
  --resource-group "${RESOURCE_GROUP}" \
  --subject "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"

We will edit the cert-manager application that is installed in the cluster.
- Go to Deployments section from the left panel and edit you application cert-manager from the Helm tab
- Pass these values in the values section and click submit
```
installCRDs: true
extraArgs:
  - --issuer-ambient-credentials
podLabels:
  azure.workload.identity/use: "true"
serviceAccount:
  labels:
    azure.workload.identity/use: "true"
```
- --issuer-ambient-credentials is a flag in cert-manager to use credentials from metadata
- The labels passed above will help cert-manager to authenticate to Azure.

Create Issuer . Make sure to create it in istio-system namespace

Fill in these variables

# your mail ID
MAIL_ID=<>

# dns zone name
DNS_ZONE_NAME="demo2.truefoundry.com"

# subscription ID
AZURE_SUBSCRIPTION_ID=<>

# get managed identity client ID
IDENTITY_CLIENT_ID=$(az identity show \
--name $IDENTITY_NAME \
--resource-group $RESOURCE_GROUP \
--query 'clientId' -otsv)

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: demo2-issuer
  namespace: istio-system
spec:
  acme:
    # give your email ID which will get notificate when the certificate gets expired
    email: $MAIL_ID
     # staging server is used for testing, it will create certificate
     # which are not publicly accepted. For valid production certificates
     # use https://acme-v02.api.letsencrypt.org/directory as acme.server
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: demo2-privkey
    solvers:
    - dns01:
        azureDNS:
          hostedZoneName: $DNS_ZONE_NAME
          resourceGroupName: $RESOURCE_GROUP
          subscriptionID: $AZURE_SUBSCRIPTION_ID
          environment: AzurePublicCloud
          managedIdentity:
            clientID: $IDENTITY_CLIENT_ID
EOF

Create a certificate

Make sure to point the issuer name from the above code correctly in issuerRef.name

Create the below certificate

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: demo2-cert
  namespace: istio-system
spec:
  # the secret in which the cert will be pasted once provisioned
  # it MUST BE PRESENT IN THE istio-system NAMESPACE
  secretName: demo2-tls
  issuerRef:
    # Issuer name from the previous step
    name: demo2-issuer
  dnsNames:
  - "demo2.truefoundry.com"
  - "*.demo2.truefoundry.com"
EOF

It will take sometime for certificate to get created

$ kubectl get cert -n istio-system
NAME         READY   SECRET      AGE
demo2-cert   True    demo2-tls   4m29

The secret demo2-tls will contain the certificates

$ kubectl get secret -n istio-system -l controller.cert-manager.io/fao=true
NAME        TYPE                DATA   AGE
demo2-tls   kubernetes.io/tls   2      5m47s

Adding the secret in ingress

Now the secret is created we need to add this secret to tfy-istio-ingress application for it terminate to SSL.
Go to Deployments section from the left panel and edit application tfy-istio-ingress .

Enter the secret in the values section at tfyGateway.spec.servers[1].tls.credentialName and click Submit

tfyGateway:
  name: tfy-wildcard
  spec:
    servers:
      - tls:
          httpsRedirect: true
        port:
          name: http-tfy-wildcard
          number: 80
          protocol: HTTP
        hosts:
          - "*.demo2.truefoundry.com"
          - "demo2.truefoundry.com"
      - port:
          name: https-tfy-wildcard
          number: 443
          protocol: HTTPS
        hosts:
          - "*.demo2.truefoundry.com"
          - "demo2.truefoundry.com"
        tls:
          mode: SIMPLE
          credentialName: demo2-tls
    selector:
      istio: tfy-istio-ingress

Once the secret is added we need to enable the base domain URL from the cluster settings.
- Go to Integrations tab and click on Edit section of the cluster.
- Add the base domain URL
Now these domain URLs can be used to expose your endpoints in the deployments section.
- Go to deployments section and create a new service (let is be nginx)
- Expose it at port 80.
- Create the service and access the URL. The format will be
```
https://nginx-<workspace-name>-80.demo2.truefoundry.com/
```