Deploy Truefoundry in an Air-gapped Environment
Overview
An air-gapped environment is isolated from internet. This means all the artifacts(helm charts, container images) needed for the platform to work have to be made available locally or in an internal network without access to the internet.
We do this by pushing all the images and helm charts to a dedicated OCI compatible container registry which should be accessible from the environment where the cluster is supposed to run. We have provided a list of images and a script to pull and push those images to any other registry.
You will also need to provide access to an npm
registry with js-yaml package hosted from within the cluster.
Requirements
In order to setup Truefoundry in an air-gapped environment, the following are needed
- An OCI compatible registry which should be able to serve the following (you can also reuse the Truefoundry's registry at
tfy.jfrog.io/tfy-images
for container images andtfy.jfrog.io/tfy-helm
for helm charts instead of replicating the images locally)- OCI compatible images with a username password or any other authentication mechanism that is suitable
- OCI compatible helm charts without authentication
- A secure environment that has access to the internet and image push access to the target registry. This is to push the images to the registry and will be required for initial setup and all upgrades
- The following should be installed in the environment to run the script
- Python
- Docker
- Helm
- Kubectl
- If running on AWS, karpenter needs access to certain APIs which have to be enabled on private VPC endpoints. Detailed instructions are available here
Uploading artifacts to a private registry
A list of images and helm charts needed by a Truefoundry installation is maintained in infra-chart repo. Here is a list of Truefoundry Helm Charts that can be deployed depending on the environment you wish to deploy Truefoundry into
- AWS EKS (tfy-k8s-aws-eks-inframold)
- Azure AKS (tfy-k8s-azure-aks-inframold)
- Civo Talos (tfy-k8s-civo-talos-inframold)
- GCP GKE Standard (tfy-k8s-gcp-gke-standard-inframold)
- Generic Kubernetes Cluster (tfy-k8s-generic-inframold)
Push images and helm charts to the Registry
- To start download the artifacts manifest file of the helm chart from the list above and save in a local file. This file contains all the container images and helm charts needed for a Truefoundry installation
wget <link_to_artifacts_manifest>
- You must login to the target registry using the
docker
CLIdocker login --username <username> --password <password> <target_registry>
- To pull and push these images to your own registry, we have provided a sample python script here. Steps to prepare the environment -
- Clone the
truefoundry/infra-charts
repogit clone https://github.com/truefoundry/infra-charts.git
- Create a virtual env to run the python script in
python3 -m venv venv source venv/bin/activate
- Install the
requirements.txt
pip install -r infra-charts/scripts/upload-artifacts/requirements.txt
- Clone the
- The
upload_artifact.py
script takes in the following argumentsartifact_type
- this takes in the artifact type which can be eitherimage
orhelm
file_path
- this is the location of theartifacts-manifest.json
file that contains the details for all the container images and helm charts that are needed for the installationdestination_registry
- this is the registry you plan to use in your air-gapped environment
- Run script with the target container registry details
python infra-charts/scripts/upload-artifacts/upload_artifacts.py \ <artifact_type> artifacts-manifest.json \ <registry_url>
Deploying Inframold Charts
- Create a namespace for jspolicy
kubectl create ns jspolicy
-
If your registry needs to be authenticated before you can pull images, then you have to authenticate the pods to pull the images. Create a secret with the registry auth information
kubectl create secret docker-registry regcred \ --docker-server=<registry_url> \ --docker-username=<username> \ --docker-password=<password> \ -n jspolicy
-
Create a file called
jspolicy-values.yaml
with the following contentimage: <registry_url>/loftsh/jspolicy:0.2.2 replicaCount: 2 env: ## Needed if npmjs.org is not accessible npm_config_registry: "https://tfy.jfrog.io/artifactory/api/npm/tfy-npm-registry-local" ## only add this section if you need to authenticate against the registry imagePullSecrets: - regcred
-
Install jspolicy helm chart
helm install jspolicy \
<registry_url>/loft/jspolicy -n jspolicy --create-namespace -f jspolicy-values.yaml
- Install the
ArgoCD
CRDs. These need to be installed first because theJSPolicy
resources reference theArgoCD
CRD
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-cd/master/manifests/crds/application-crd.yaml
-
Install
tfy-jspolicy-config
Chart to patch your images-
Create a file called
jspolicy-config-values.yaml
with the following content. Make sure to add the registry url with proper protocol forreplaceArgoHelmRepo.registryReplacementMap
## @section replaceImageRegistry Configuration options for replacing the image registry replaceImageRegistry: ## @param replaceImageRegistry.enabled Enable or disable replacing the image registry enabled: true ## @param replaceImageRegistry.excludeNamespaces Namespaces to exclude from replacing the image registry excludeNamespaces: - kube-system - kube-public - kube-node-lease ## @param replaceImageRegistry.includeNamespaces Namespaces to include from replacing the image registry. When non-empty, only these namespaces will be included includeNamespaces: [] ## @param replaceImageRegistry.registryReplacementMap The image registry replacement map ## registryReplacementMap: ## "docker.io": "mydocker.io" ## "*": "myjrog.io" registryReplacementMap: "*": <registry_url> ## @section replaceArgoHelmRepo Configuration options for replacing the Argo Helm repository replaceArgoHelmRepo: ## @param replaceArgoHelmRepo.enabled Enable or disable replacing the Argo Helm repository enabled: true ## @param replaceArgoHelmRepo.excludeNamespaces Namespaces to exclude from replacing the argo helm repo excludeNamespaces: [] ## @param replaceArgoHelmRepo.includeNamespaces Namespaces to include for replacing the argo helm repo. When non-empty, only these namespaces will be included includeNamespaces: [] ## @param replaceArgoHelmRepo.registryReplacementMap The argo helm repository replacement map. Protocol is mandatory ## registryReplacementMap: ## oci://myargohelm.io: "oci://tfyargohelm.io" ## http://myargohelm.io: "http://tfyargohelm.io" registryReplacementMap: "*": <registry_url>
-
Install the tfy-jspolicy-config chart
helm install tfy-jspolicy-config <registry_url>/infra-charts/tfy-jspolicy-config --version 0.1.5 -n jspolicy -f jspolicy-config-values.yaml
-
-
Now the cluster is ready and you can continue with the appropriate installation steps from here
Updated 2 months ago