Mounting Secrets from AWS Systems Manager Parameter Store for AWS EKS

2023-02-11

🎯 Objectives

  • Mounting Secrets from AWS Systems Manager Parameter Store for AWS EKS

Why Parameter Store ?

  • Leverages AWS KMS to encrypt values.
  • Support versioning of secret values
  • It is free and no additional charge for storage and standard throughput. For higher throughput, API interactions cost is $0.05 per 10,000 API calls.
  • You want cheaper option to store encrypted or unencrypted secrets.

Prerequisite

  • An AWS account and AWS CLI
  • An existing EKS Cluster
  • Helm, Docker, and Kubectl installation.
  1. Export variables
1
2
3
4
5
6
7
8
export AWS_ACCESS_KEY_ID="ASIAV....."
export AWS_SECRET_ACCESS_KEY="XtRzQJ20fGFAil....."
export AWS_REGION=<preferred_region>
export EKS_CLUSTER_NAME=<cluster_name>
export IAM_POLICY_NAME=<preferred_name>
export NAMESPACE=<your_k8s_namespace>
export SERVICE_ACCOUNT_NAME=<k8s_service_acount>
export IAM_ROLE_NAME=<preferred_name>
  1. Install the AWS Secrets and Config Provider (ASCP) and Container Storage Interface (CSI) driver
1
2
3
4
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm  upgrade -install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
 --set syncSecret.enabled=true --set enableSecretRotation=true
kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
  1. Create IAM policy with permissions that allow your your pods to access the parameters in Parameter Store
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
cat > parameter-policy.json <<EOF

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:GetSecretValue",
                "ssm:DescribeParameters",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:GetParametersByPath"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "kms:DescribeCustomKeyStores",
                "kms:ListKeys",
                "kms:ListAliases"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:GetKeyRotationStatus",
                "kms:GetKeyPolicy",
                "kms:DescribeKey"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}
EOF
aws iam create-policy --policy-name $IAM_POLICY_NAME --policy-document file://parameter-policy.json
  1. Create an IAM role:

i. View your cluster’s OIDC provider URL

1
aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text

ii. Create the following IAM trust policy file , replace YOUR_AWS_ACCOUNT_ID with your account ID. Replace YOUR_AWS_REGION with your AWS Region. Replace XXXXXXXXXX45D83924220DC4815XXXXX with the value returned in step 4i.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cat <<EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
           "oidc.eks.eu-west-1.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>:aud": "sts.amazonaws.com",
          "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>:sub": "system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT_NAME"
        }
      }
    }
  ]
}
EOF

iii. Create an IAM role

1
2
3
aws iam create-role \
  --role-name $IAM_ROLE_NAME \
  --assume-role-policy-document file://trust-policy.json

iv. Attach your new IAM policy created in step 3 to the role:

1
2
3
aws iam attach-role-policy \
--policy-arn arn:aws:iam::YOUR_AWS_ACCOUNT_ID:policy/$IAM_POLICY_NAME \
--role-name $IAM_ROLE_NAME
  1. Annotate the Kubernetes service account with the Amazon Resource Name (ARN) of the IAM role that you created earlier and replace with your AWS_ACCOUNT_ID:
1
2
3
4
5
6
7
8
9
10
11
12
cat > service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: $SERVICE_ACCOUNT_NAME
  namespace: $NAMESPACE
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ACCOUNT_ID:role/$IAM_ROLE_NAME
EOF

kubectl apply -f service-account.yaml

  1. Create fake parameters , I will create a fake staging postgresql DB username and password
1
2
3
4
5
6
aws ssm put-parameter --name "/staging/postgresql/username" --value "admin" --description "username for staging postgreSQL DB" \
 --type SecureString --region $AWS_REGION

aws ssm put-parameter --name "/staging/postgresql/password" --value "4g#4gGGDG9OghjuE" --description "password for staging postgreSQL DB" \
 --type SecureString --region $AWS_REGION

  1. Create SecretProviderClass and Sync with Native Kubernetes Secrets
    Create SecretProviderClass custom resource with provider:aws The SecretProviderClass must be in the same namespace as the pod using it later.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
cat << EOF > k8s-secrets.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: staging-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "/staging/postgresql/username"
        objectType: "ssmparameter"
        objectAlias: postgresUserAlias
      - objectName: "/staging/postgresql/password"
        objectType: "ssmparameter"
        objectAlias: postgresPasswordAlias


  secretObjects:
    - secretName: staging-env
      type: Opaque
      data:
        - objectName: postgresUserAlias
          key: postgres_username
        - objectName: postgresPasswordAlias
          key: postgres_password
EOF
kubectl apply -f  k8s-secrets.yaml -n $NAMESPACE

  1. Create pod , mount secrets volumes and set up Environment variables
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
cat << EOF > deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: login-app
  labels:
    app: login-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: login-app
  template:
    metadata:
      labels:
        app: login-app
    spec:
      serviceAccountName: $SERVICE_ACCOUNT_NAME
      containers:
      - name: login-app
        image: <your-image-name>
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 80
        volumeMounts:
          - name: secrets-store-inline
            mountPath: "/mnt/secrets"
            readOnly: true
        env:
          - name: POSTGRE_USERNAME
            valueFrom:
              secretKeyRef:
                name: staging-env
                key: postgres_username
          - name: POSTGRE_PASSWQRD
            valueFrom:
              secretKeyRef:
                name: staging-env
                key: postgres_password
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: staging-secrets
EOF
kubectl apply -f deployment.yaml -n $NAMESPACE

References

  1. AWS EKS Workshop
  2. AWS Docs
  3. HANDLING SECRETS AND PARAMETERS ON AWS EKS by Charles Guebels

Hello 👋, If you enjoyed this article, please consider subscribing to my email newsletter. Subscribe 📭