Managing Kubernetes Secrets with AWS Secrets Manager

In the last part of this series, I introduced GoDaddy’s Kubernetes External Secrets Manager. In this installment, we will leverage it to configure secrets backed by Amazon Web Services‘ Secrets Manager.

GoDaddy extensively relies on Amazon Web Services‘ EKS for running their Kubernetes infrastructure. The engineering team at GoDaddy realized that there is no integration between EKS and other managed services like Amazon Secrets Manager and AWS Systems Manager. To bridge the gap between the two, they built a Kubernetes custom controller and a custom resource definition called External Secrets, which can manage and rotate the keys originating in AWS Secrets Manager for any Kubernetes deployment.

Let’s see Kubernetes External Secrets in action.

Prerequisites:

1. Active AWS Subscription
2. AWS CLI
3. ​​Kubernetes Cluster

We will start by creating an AWS Identity Access Management (IAM) Policy and an AWS IAM User with just enough permissions to read the secrets from AWS Secrets Store. In the next step, we will deploy the Kubernetes External Secrets Manager Helm Chart associated with the IAM User credentials. After that, we create a secret in the AWS Secrets Store and pointing it to the Kubernetes External Secret created in the local cluster. Finally, we update the cloud-based secret and verify if it’s refreshed in the Kubernetes cluster.

Step 1: Create an AWS IAM Policy and IAM User to Access Secrets Store

Let’s start by defining the IAM Policy needed to access the secrets. Create a JSON file with the below content and save it in extsecpol.json file.

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "secretsmanager:GetRandomPassword",
 "secretsmanager:GetResourcePolicy",
 "secretsmanager:GetSecretValue",
 "secretsmanager:DescribeSecret",
 "secretsmanager:ListSecretVersionIds",
 "secretsmanager:ListSecrets"
 ],
 "Resource": "*"
 }
 ]
}

Define the environment variables and proceed with the creation of IAM Policy and Role.

POLICYNAME="ExternalSecurityPolicy"
IAMUSERNAME="SecretReader"

aws iam create-policy \
	--policy-name $POLICYNAME \
	--policy-document file://extsecpol.json

Retrieve and store the Amazon Resource Name (ARN) for the policy in an environment variable.

export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`ExternalSecurityPolicy`].{ARN:Arn}' --output text)

Make sure you replace ExternalSecurityPolicy with the correct policy name.

Create the IAM User based on the above policy.

aws iam create-user --user-name $IAMUSERNAME
aws iam attach-user-policy --user-name $IAMUSERNAME --policy-arn $POLICYARN
aws iam create-access-key --user-name $IAMUSERNAME

From the output of the last command, retrieve the Access Key and Secret Key associated with the user.

👁 Image

Step 2: Install Kubernetes External Secrets Helm Chart

Before we deploy the Kubernetes External Secrets custom controller, we need to register the AWS AccessKey and SecretAccessKey as Kubernetes secrets in the same namespace where the custom controller runs.

Let’s start by creating the namespace.

kubectl create namespace external-secrets

Encode AccessKey and SecretAccessKey into base64 and turn them into a Kubernetes secret.

echo -n "AKIATWRVOGK5V3ZITMWO" | base64
echo -n "6nGIROhIkZGFFvvhkTFWefX5ONFCZyvZZdTIDllR" | base64

Create a file named aws-secrets.yaml with the base64-encoded values and apply it with kubectl command.

apiVersion: v1
kind: Secret
metadata:
 name: aws-credentials
type: Opaque
data:
 id: QUtJQVRXUlZPR0s1VjNaSVRNV08=
 key: Nm5HSVJPaElrWkdGRnZ2aGtURldlZlg1T05GQ1p5dlpaZFRJRGxsUg==

kubectl apply -f aws-secrets.yaml -n external-secrets

Let’s associate the secret with the Helm Chart by defining the below values.yaml file:

env:
 AWS_REGION: ap-south-1
 AWS_DEFAULT_REGION: ap-south-1

envVarsFromSecret:
 AWS_ACCESS_KEY_ID:
 secretKeyRef: "aws-credentials"
 key: "id"
 AWS_SECRET_ACCESS_KEY:
 secretKeyRef: "aws-credentials"
 key: "key"

Replace the AWS_REGION with the region where you created the secret.

Now, we are ready to install the Kubernetes External Secrets Helm Chart.

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
helm repo update

helm install \
	--generate-name \
	--namespace external-secrets \
	external-secrets/kubernetes-external-secrets \
	--values values.yaml

Verify the deployment and ensure there are no errors.

kubectl get pods -n external-secrets

👁 Image

Step 3: Create a Secret in AWS Secrets Manager

With the Kubernetes cluster connected to the AWS Secrets Manager through the External Secrets controller, we are ready to create secrets in the cloud and propagate them to the local cluster.

aws secretsmanager create-secret \
	--region ap-south-1 \
	--name edgesecrets/dbcred \
	--secret-string '{"username":"admin","password":"password@1234"}'

👁 Image

In the next step, we will create an External Secret pointed to the secret created in the AWS Secrets Manager.

Step 4: Create an External Secret Resource in Kubernetes

Let’s create an External Secret custom resource called dbcred associated with the cloud-based secret created in the previous step in the default namespace.

apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
 name: dbcred
spec:
 backendType: secretsManager
 region: ap-south-1
 dataFrom:
 - edgesecrets/dbcred

kubectl create -f dbcred-external.yaml

Verify that the External Secret and the Kubernetes secret are created successfully.

kubectl get es

👁 Image

kubectl get secrets

👁 Image

Let’s retrieve the values of the secret by decoding the base64 values.

kubectl get secret dbcred -o yaml

👁 Image

echo " YWRtaW4=" | base64 -d

echo "cGFzc3dvcmRAMTIzNA==" | base64 -d

👁 Image

Access the AWS Management Console to retrieve the secret values.

👁 Image

As you can see, the same values showed in the AWS Console are available in the Kubernetes cluster.

Step 5: Rotating/Refreshing the Secrets

Let’s now modify the secret stored on AWS Secrets Store to see if it automatically gets propagated to the Kubernetes cluster.

Run the below command to change the password to a new value:

aws secretsmanager update-secret \
 --region ap-south-1 \
 --secret-id edgesecrets/dbcred \
 --secret-string '{"username":"admin","password":"password@0000"}'

Retrieve the password value from the Kubernetes cluster.

kubectl get secret dbcred -o yaml

👁 Image

echo "cGFzc3dvcmRAMDAwMA==" | base64 -d

👁 Image

As we can see, the updated password instantly becomes available in the Kubernetes cluster.

Step 6: Cleaning up Resources

Delete Kubernetes resources:

kubectl delete -f dbcred-external.yaml
helm ls -n external-secrets
helm delete RELEASE_NAME
kubectl delete -f aws-secrets.yaml -n external-secrets

Delete AWS IAM Policy:

IAMUSERNAME=SecretReader
export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`ExternalSecurityPolicy`].{ARN:Arn}' --output text)
aws iam detach-user-policy --user-name $IAMUSERNAME --policy-arn $POLICYARN

Delete AWS IAM User:

export ACCESSKEYID=$(aws iam list-access-keys --user-name SecretReader --query 'AccessKeyMetadata[0].AccessKeyId' --output text)
aws iam delete-access-key --user-name $IAMUSERNAME --access-key-id $ACCESSKEYID

aws iam delete-user --user-name $IAMUSERNAME
aws iam delete-policy --policy-arn $POLICYARN

Delete AWS Secret Manager secret:

aws secretsmanager delete-secret --secret-id edgesecrets/dbcred