Deploy Truefoundry in an Air-gapped Environment
Overview
An air-gapped environment simply means an environment that has been isolated from other environments or networks or more commonly from the internet.
When deploying Truefoundry components, by default there would be a need to access the internet to get the artifacts (Images and Helm Charts) that will be needed for a successful deployment. Due to the nature of an air-gapped systems, it will not be possible to get these artifacts. In order to circumvent this, we have come up with a way to get the necessary artifacts into your systems or environmnets; The following steps will explain how this will be done or achieved.
Deploying Inframold charts from a private registry would require having the necessary artifacts required to successfully deploy the chart to be available in the registry. In order to achieve this, we have a manifest file for each charts that details the required artifacts needed and a script that makes it easy to deploy those artifacts to your own private registry.
Requirements
In order to be setup Truefoundry in an air-gapped environment, the following are needed
- An OCI compatible registry which is accessible by the cluster.
- The registry should also be able to serve helm charts from an unauthenticated endpoint.
- A secure environment that has access to the internet and can also upload artifacts to the secure registry. This is for initial setup.
- The following should also be install in the environment;
- Docker
- Helm
- Kubectl
Uploading artifacts to a private registry
Deploying Inframold charts from a private registry would require having the necessary artifacts required to successfully deploy the chart to be available in the registry. In order to achieve this, we have an artifact-manifest file which details the required artifacts needed for each Truefoundry chart and a script that makes it easy to deploy these artifacts to your own private registry. The infra-chart repo contains a list of 3rd party chart dependencies alongside Truefoundry helm charts that can be deployed, here is a list of Truefoundry Helm Charts that can be deployed depending on the environment you wish to deploy Truefoundry into;
- AWS EKS (tfy-k8s-aws-eks-inframold)
- Azure AKS (tfy-k8s-azure-aks-inframold)
- Civo Talos (tfy-k8s-civo-talos-inframold)
- GCP GKE Autopilot (tfy-k8s-gcp-gke-autopilot-inframold)
- GCP GKE Standard (tfy-k8s-gcp-gke-standard-inframold)
- Generis Kubernetes Cluster (tfy-k8s-generic-inframold)
Steps to deploy artifacts to a private registry
- To start download the artifacts manifest file of the helm chart from the list above, python script and requirements.txt files locally
- Create a python virtual environment so as to have an isolated environment for the upload_artifact.py script. You can follow the virtualenv guide for installation.
- Install the python dependencies
pip install -r requirements.txt
- The
upload_artifact.py
is a sample python scripts which handles deployment to JFrog and ECR registries, it takes in the following arguments;artifact_type
- this takes in the artifact type which can be eitherimage
orhelm
file_path
- this is the location of the artefacts-manifest.json file that contains the artefacts information.destination_registry
- this is the registry you plan to use in your air-gapped environment.registry_type
the type of registry. The script currently take ecr for AWS registry and jfrog
- Before uploading images or charts to your registry, make sure you're authenticated. When setting up jfrog, you have to create a token the script will use. Follow this link to generate your token. Once you get the token, pass it in as an environment variable -
JFROG_TOKEN
before running the script.
When deploying to ECR, the following are needed to be passed as environment variables -AWS_PROFILE
andAWS_REGION
.
To update your registry with the required images to a JFROG registry, run the following command
python upload_artifacts.py image artifacts-manifest.json <jfrog-registry-tenant>.jfrog.io jfrog
python upload_artifacts.py image artifacts-manifest.json <jfrog-registry-tenant>.jfrog.io jfrog
Deploying Inframold Charts
- Create a namespace for kyverno
kubectl create ns kyverno
-
If your registry needs to be authenticated before you can pull images, then you have to authenticate the pods to pull the images. Create a secret with the registry auth information
kubectl create secret docker-registry regcred \ --docker-server=tfyregistry.jfrog.io \ --docker-username="" \ --docker-password="" \ -n kyverno
-
Create a file called
kyverno-values.yaml
with the following contentglobal: image: registry: tfyregistry.jfrog.io/truefoundry ## only add this section if you need to authenticate against the registry existingImagePullSecrets: - regcred webhooksCleanup: imagePullSecrets: - name: regcred policyReportsCleanup: imagePullSecrets: - name: regcred # Cleanup cronjobs to prevent internal resources from stacking up in the cluster cleanupJobs: imagePullSecrets: - name: regcred # Cluster admission reports configuration clusterAdmissionReports: imagePullSecrets: - name: regcred # Admission controller configuration admissionController: imagePullSecrets: - name: regcred # Background controller configuration backgroundController: imagePullSecrets: - name: regcred # Cleanup controller configuration cleanupController: imagePullSecrets: - name: regcred # Reports controller configuration reportsController: imagePullSecrets: - name: regcred
-
Install kyverno helm chart
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno --version 3.2.5 -n kyverno -f kyverno-values.yaml
- Install argocd crds, this needs to be installed first because the Kyverno ClusterPolicy references the ArgoCD CRD
kubectl apply -f https://github.com/argoproj/argo-cd/manifests/crds\?ref\=stable
-
Install Kyverno Config Chart to patch your images
-
Create a file called kyverno-config.yaml with the following content
## @section replaceImageRegistry Configuration options for replacing the image registry replaceImageRegistry: ## @param replaceImageRegistry.enabled Enable or disable replacing the image registry enabled: true ## @param replaceImageRegistry.newRegistry The new image registry to use newRegistry: "<registry-url>" ## @section replaceArgoHelmRepo Configuration options for replacing the Argo Helm repository replaceArgoHelmRepo: ## @param replaceArgoHelmRepo.enabled Enable or disable replacing the Argo Helm repository enabled: true ## @param replaceArgoHelmRepo.newRegistry The new Argo Helm registry URL newRegistry: "<regsitry-url>"
-
Install the kyverno-config chart
helm repo add truefoundry https://truefoundry.github.io/infra-charts/ helm repo update helm install kyverno-config truefoundry/tfy-kyverno-config --version 0.1.3 -n kyverno -f kyverno-config.yaml
-
-
Install argoCD
kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/core-install.yaml
-
We will create a values.yaml for the helm chart installation
- Download the values.yaml from helm chart repo
curl https://raw.githubusercontent.com/truefoundry/infra-charts/main/charts/tfy-k8s-aws-eks-inframold/values-cp.yaml > values.yaml
- Fill in the
tenant_name
,cluster_name
,truefoundry_image_pull_config_json
andtfy_api_key
in the downloaded file. You can get these from the Truefoundry team. - Also fill in the database section with database creds. If you created the infrastructure using OCLI, you can get the credentials by running ocli output
- Download the values.yaml from helm chart repo
-
Apply the helm chart with the values.yaml
helm install -n argocd inframold truefoundry/tfy-k8s-aws-eks-inframold --version 0.0.16 -f values.yaml
Updated about 1 month ago