Setting Up BackUp and Disaster Recovery on AWS EKS with Velero

2023-04-16

🎯 Objectives

  • Set up backup and restore, perform disaster recovery, migrate Kubernetes cluster resources and persistent volumes with Velero.

Prerequisite

  • An AWS account
  • Set up 2 EKS Clusters ( Primary and Recovery). The first cluster should have deployed applications, while the other cluster should be empty.
  • Create velero namespace on both clusters.
  • Helm, Docker , eksctl and Kubectl installation.

Set up Velero

Export AWS credentials and other environment variables
1
2
3
4
5
6
7
8
9
10

export AWS_ACCESS_KEY_ID="ASIAV....."
export AWS_SECRET_ACCESS_KEY="XtRzQJ20fGFAil....."
BUCKET=<your-bucket-name>
REGION=<your-preferred-aws-region>
PRIMARY_CLUSTER=<primary_cluster_name>
RECOVERY_CLUSTER=<recovery_cluster_name>
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RELEASE_NAME=<helm-release-name>
NAMESPACE=velero
Create a new bucket in your desired region
1
aws s3 mb s3://$BUCKET --region $REGION
Create IAM policy file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

cat > velero_policy.json <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "ec2:DescribeVolumes",
               "ec2:DescribeSnapshots",
               "ec2:CreateTags",
               "ec2:CreateVolume",
               "ec2:CreateSnapshot",
               "ec2:DeleteSnapshot"
           ],
           "Resource": "*"
       },
       {
           "Effect": "Allow",
           "Action": [
               "s3:GetObject",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:AbortMultipartUpload",
               "s3:ListMultipartUploadParts"
           ],
           "Resource": [
               "arn:aws:s3:::${BUCKET}/*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3:::${BUCKET}"
           ]
       }
   ]
}
EOF

aws iam create-policy \
   --policy-name VeleroAccessPolicy \
   --policy-document file://velero_policy.json

Create Service Accounts for Velero

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
eksctl create iamserviceaccount \
--cluster=$PRIMARY_CLUSTER \
--name=velero-server \
--namespace=$NAMESPACE \
--role-name=eks-velero-backup \
--role-only \
--attach-policy-arn=arn:aws:iam::$ACCOUNT:policy/VeleroAccessPolicy \
--approve

eksctl create iamserviceaccount \
--cluster=$RECOVERY_CLUSTER \
--name=velero-server \
--namespace=$NAMESPACE \
--role-name=eks-velero-recovery \
--role-only \
--attach-policy-arn=arn:aws:iam::$ACCOUNT:policy/VeleroAccessPolicy \
--approve

Set up Velero and it’s Helm Chart on your local system

1
2
3
4
5
wget https://github.com/vmware-tanzu/velero/releases/download/v1.2.0/velero-v1.2.0-linux-amd64.tar.gz
tar -zxvf velero-v1.2.0-linux-amd64.tar.gz
cp velero-v1.2.0-linux-amd64/velero /usr/local/bin
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

Install Velero in Primary Cluster and Perform Automatic Scheduling on a single namespace

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

cat > values.yaml <<EOF
configuration:
  backupStorageLocation:
    bucket: $BUCKET
  provider: aws
  volumeSnapshotLocation:
    config:
      region: $REGION
credentials:
  useSecret: false
initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.2.0
  volumeMounts:
  - mountPath: /target
    name: plugins
serviceAccount:
  server:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::${ACCOUNT}:role/eks-velero-backup"
EOF

aws eks update-kubeconfig --name $PRIMARY_CLUSTER --region $REGION
helm upgrade -f values.yaml  --install $RELEASE_NAME vmware-tanzu/velero --namespace $NAMESPACE
velero schedule create <desired-backup-name> --schedule="0 19 * * 5" --include-namespaces <namespace-with-deployed-k8s-object> --ttl 720h0m0s
velero get schedule

Note You can use this cron tab simulator to design the scheduling that works for you

Install Velero in Recovery Cluster and Restore Backup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

cat > values_recovery.yaml <<EOF
configuration:
  backupStorageLocation:
    bucket: $BUCKET
  provider: aws
  volumeSnapshotLocation:
    config:
      region: $REGION
credentials:
  useSecret: false
initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.2.0
  volumeMounts:
  - mountPath: /target
    name: plugins
serviceAccount:
  server:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::${ACCOUNT}:role/eks-velero-recovery"
EOF


aws eks update-kubeconfig --name $RECOVERY_CLUSTER --region $REGION
helm upgrade -f values_recovery.yaml  --install $RELEASE_NAME vmware-tanzu/velero --namespace $NAMESPACE

velero restore create <desired-recovery-name> \
    --from-backup <desired-backup-name> \
    --include-namespaces <namespace-with-deployed-k8s-object>

Hello 👋, If you enjoyed this article, please consider subscribing to my email newsletter. Subscribe 📭