Documentation

EFS in AWS EKS

Setting up EFS support in your EKS cluster

Suggest Edits

This section describe how you can achieve EFS support in your EKS cluster. EFS persistent volume claims go across availability zones and support RWX modes.

Pre-requisites

  1. Authenticate to aws cli.
  2. Set these values before running the below commands 
    export CLUSTER_NAME=""
export AWS_REGION=""
export ACCOUNT_ID=""
  3. Get the cluster OIDC issuer URL. If you don't have sed installed then manually remove the https:// from the OIDC by running the command without the pipe and sed 
    OIDC_ISSUER_URL=$(aws eks describe-cluster --name $CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed 's/https:\/\///')
  4. Get the VPC ID in which. you cluster is running 
    # get the VPC ID
VPC_ID=$(aws eks describe-cluster \
    --name "${CLUSTER_NAME}" \
    --query "cluster.resourcesVpcConfig.vpcId" \
    --region "${AWS_REGION}" \
    --output text)
    
# get the CIDR range
VPC_CIDR_RANGE=$(aws ec2 describe-vpcs \
    --vpc-ids $vpc_id \
    --query "Vpcs[].CidrBlock" \
    --output text \
    --region "${AWS_REGION}")

Setting up IAM policy and roles

  1. Create the below IAM policy
    1. Download the IAM policy 
      curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json
    2. Create the policy 
      aws iam create-policy \
    --policy-name "EFS_CSI_DRIVER_POLICY-${CLUSTER_NAME}" \
    --policy-document file://iam-policy-example.json
  2. We will create a role which has the above policy.
    1. Create the file by running the below command. This allows EFS CSI driver pods to use their serviceaccount to talk to EFS. 
      cat > trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_ISSUER_URL}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_ISSUER_URL}:sub": "system:serviceaccount:aws-efs-csi-driver:efs-csi-controller-sa"
        }
      }
    }
  ]
}
EOF
    2. Create the role with the above trust relationship. 
      EFS_ROLE_ARN=$(aws iam create-role \
  --role-name "EFS_CSI_DRIVER_ROLE-${CLUSTER_NAME}" \
  --assume-role-policy-document file://"trust-policy.json" \
  --query 'Role.Arn' --output text)
    3. Attach the policy that we created in Step 1 to the role 
      aws iam attach-role-policy \
  --policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/EFS_CSI_DRIVER_POLICY-${CLUSTER_NAME}" \
  --role-name "EFS_CSI_DRIVER_ROLE-${CLUSTER_NAME}"

Creating AWS EFS

  1. Creating a security group to allow 2049 port access from VPC.

    1. Create a security group 
      SECURITY_GROUP_ID=$(aws ec2 create-security-group \
    --group-name TfyEfsSecurityGroup \
    --description "Truefoundry EFS security group" \
    --vpc-id "${VPC_ID}" \
    --region "${AWS_REGION}" \
    --output text)
    2. Authorise the security group for nodes in vpc to connect to EFS at port 2049. You can customise this security group to make it more restrictive by using the subnet CIDR where your nodes are running instead of VPC CIDR. 
      aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP_ID \
    --protocol tcp \
    --port 2049 \
    --region "${AWS_REGION}" \
    --cidr "${VPC_CIDR_RANGE}"

  2. Create the file system 

    FILE_SYSTEM_ID=$(aws efs create-file-system \
    --region "${AWS_REGION}" \
    --performance-mode generalPurpose \
    --encrypted \
    --throughput-mode elastic \
    --tags Key=Name,Value="${CLUSTER_NAME}-efs" Key=Created-By,Value=Truefoundry Key=cluster-name,Value=$CLUSTER_NAME \
    --query 'FileSystemId' \
    --output text)

  3. Create the mount targets for subnets in your nodes

    1. Capture all your subnets in the below list and run the for loop command to mount the EFS in all private subnets. Through this all your nodes will be able to mount EFS. Run the below command to get all the subnets in your VPC ID 

      aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=${VPC_ID}" \
    --query 'Subnets[*].{SubnetId: SubnetId,AvailabilityZone: AvailabilityZone,CidrBlock: CidrBlock}' \
    --output table

    2. Command to mount target 

      for subnet_id in {<LIST OF SUBNETS COMMA SEPERATED>}
do
aws efs create-mount-target \
    --file-system-id "${FILE_SYSTEM_ID}" \
    --subnet-id $subnet_id \
    --security-groups "${SECURITY_GROUP_ID}" \
    --region "${AWS_REGION}"
done

################# EXAMPLE ########################
#for subnet_id in {"subnet-09842fe267b586972","subnet-030f97cf00b7a9459"}
#do
#aws efs create-mount-target \
#    --file-system-id "${FILE_SYSTEM_ID}" \
#    --subnet-id $subnet_id \
#    --security-groups "${SECURITY_GROUP_ID}" \
#    --region "${REGION}"
#done
#################################################

Installing AWS EFS CSI Driver

  1. Once the file system is created we need to install the driver in our kubernetes cluster so that it can create volumes for EFS on our behalf

  2. From the Integrations tab , click on Manage Applications tab from the three dots on your cluster card.

  3. Install Aws Efs Csi Driver which will first prompt for workspace creation and then will ask to confirm the applications installation.

  4. Fill you region and the Role ARN which was created while Setting up IAM policy and roles 

    image:
  repository: 602401143452.dkr.ecr.$AWS_REGION.amazonaws.com/eks/aws-efs-csi-driver
controller:
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: "${EFS_ROLE_ARN}"

  5. Once the applications is installed a storage class needs to be created for persistent volume to use.

Installing a storage class and testing EFS volumes

  1. Create a storage with the FILE_SYSTEM_ID that we created 
    kubectl apply -f -<<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: "${FILE_SYSTEM_ID}"
  directoryPerms: "700"
  gidRangeStart: "1000" # optional
  gidRangeEnd: "2000" # optional
  basePath: "/truefoundry"
EOF
  2. Check if the storage class is created by the name efs-sc 
    kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
efs-sc          efs.csi.aws.com         Delete          Immediate              false                  20s
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  99d
  3. Deploy a sample pod which will create a pvc and pod using that pvc. If pod comes up successfully then EFS is working fine 
    kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/pod.yaml

References

Updated 3 months ago