Skip to main content

​
Overview

By default, Amazon VPC CNI will assign Pods an IP address selected from the primary subnet. The primary subnet is the subnet CIDR that the primary ENI is attached to, usually the subnet of the node/host. If the subnet CIDR is too small, the CNI may not be able to acquire enough secondary IP addresses to assign to your Pods. This is a common challenge for EKS IPv4 clusters. Custom networking is one solution to this problem and is supported by Truefoundry. For more details, please refer to the Amazon EKS Custom Networking documentation.

​
Setup

To setup custom networking in EKS, you need to follow the steps below:
  1. Attach a secondary CIDR to the VPC and create new subnets with the secondary CIDR in the same AZs as the primary subnet. For example, if the primary subnet is 10.0.0.0/16, you can attach a secondary CIDR of 100.64.0.0/16 (RFC6598) to the VPC and if you have primary subnets in 3AZs which are used for the EKS nodes, you have to create 3 new subnets with the secondary CIDR in the same AZs.
  2. Ensure that secondary subnets are added to the route tables of your primary subnets. This should happen automatically when you create the subnets.
  3. We need to make the AWS VPC CNI aware of the secondary subnets. If you have used TrueFoundry’s terraform code to setup your cluster, you need to make the following change in the EKS module
cluster_addons_vpc_cni_additional_configurations = {
    env = {
      AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG = "true"
      ENI_CONFIG_LABEL_DEF               = "topology.kubernetes.io/zone"
    }
}
  1. In all other cases you can set the same env variable by running the following commands from the kubernetes cli:
kubectl set env daemonset aws-node -n kube-system "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true"
kubectl set env daemonset aws-node -n kube-system "ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone"
  1. Create the following ENIConfig resources in the cluster for each AZ of the EKS nodes
apiVersion : crd.k8s.amazonaws.com/v1alpha1
kind : ENIConfig
metadata:
  name: us-east-1a # us-east-1a is the AZ of the EKS nodes
spec:
  securityGroups:
    - sg-0dff111a1d11c1c11 # sg-0dff111a1d11c1c11 is the security group of the EKS nodes
  subnet: subnet-011b111c1f11fdf11 # subnet-011b111c1f11fdf11 is the subnet with the secondary CIDR
  1. Restart the nodes one by one to apply the changes in the cluster networking. Pods will be rescheduled to the new nodes with the secondary IP addresses.