Real incidents:
- Uber: $148M fine (credentials in GitHub)
- Capital One: 100M records (misconfigured IAM)
- Codecov: Supply chain attack (exposed secrets)
- Travis CI: Exposed environment variables

Average cost of data breach: $4.35M (IBM 2023)

# NEVER do this
AWS_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
DB_PASSWORD = "supersecret123"

# This is searchable on GitHub
cat .env
DATABASE_URL=postgresql://user:password@localhost/db
API_KEY=sk_live_51H...
STRIPE_SECRET=whsec_...

# Error messages often dump environment
import os
print(os.environ) # All secrets exposed in logs

# Container inspect shows env vars
docker inspect <container> | grep -i password

# Kubernetes Secrets are only base64 encoded, not encrypted
kubectl get secret db-password -o yaml
# Anyone with cluster access can decode

Application → Vault Agent → Vault Server → Storage Backend
 ↓
 (Encrypted)
 ↓
 Consul/etcd/S3

# Key-Value secrets (static)
vault kv put secret/database/config \
 username="dbuser" \
 password="supersecret"

# Dynamic secrets (generated on-demand)
vault read database/creds/readonly
# Returns temporary credentials that auto-expire

# Kubernetes authentication
vault write auth/kubernetes/role/myapp \
 bound_service_account_names=myapp \
 bound_service_account_namespaces=production \
 policies=myapp-policy \
 ttl=1h

# myapp-policy.hcl
path "secret/data/database/config" {
 capabilities = ["read"]
}

path "database/creds/readonly" {
 capabilities = ["read"]
}

path "secret/data/api-keys/*" {
 capabilities = ["read", "list"]
}

# Vault on Kubernetes with Helm
helm repo add hashicorp https://helm.releases.hashicorp.com

helm install vault hashicorp/vault \
 --set server.ha.enabled=true \
 --set server.ha.replicas=3 \
 --set ui.enabled=true \
 --set server.dataStorage.size=10Gi

# Initialize Vault
kubectl exec vault-0 -- vault operator init
# Save unseal keys and root token securely!

# Unseal Vault (repeat on all pods)
kubectl exec vault-0 -- vault operator unseal <unseal-key-1>
kubectl exec vault-0 -- vault operator unseal <unseal-key-2>
kubectl exec vault-0 -- vault operator unseal <unseal-key-3>

apiVersion: apps/v1
kind: Deployment
metadata:
 name: myapp
spec:
 template:
 metadata:
 annotations:
 vault.hashicorp.com/agent-inject: "true"
 vault.hashicorp.com/role: "myapp"
 vault.hashicorp.com/agent-inject-secret-database: "secret/data/database/config"
 vault.hashicorp.com/agent-inject-template-database: |
 {{- with secret "secret/data/database/config" -}}
 DATABASE_URL=postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/mydb
 {{- end }}
 spec:
 serviceAccountName: myapp
 containers:
 - name: app
 image: myapp:v1.0
 # Secret automatically written to /vault/secrets/database

import hvac

# Authenticate with Kubernetes
client = hvac.Client(url='http://vault:8200')

with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
 jwt = f.read()

client.auth.kubernetes.login(
 role='myapp',
 jwt=jwt
)

# Read secrets
secret = client.secrets.kv.v2.read_secret_version(
 path='database/config'
)

db_user = secret['data']['data']['username']
db_pass = secret['data']['data']['password']

# Configure database secrets engine
vault secrets enable database

vault write database/config/postgresql \
 plugin_name=postgresql-database-plugin \
 allowed_roles="readonly" \
 connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb" \
 username="vault" \
 password="vaultpass"

# Create role for read-only access
vault write database/roles/readonly \
 db_name=postgresql \
 creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
 GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
 default_ttl="1h" \
 max_ttl="24h"

# Application requests credentials
vault read database/creds/readonly
# Returns:
# Key Value
# lease_id database/creds/readonly/abc123
# lease_duration 1h
# username v-readonly-abc123
# password A1a-generated-password

# Credentials automatically revoked after 1h

# Automatic rotation for supported systems
vault write -f database/rotate-root/postgresql
# Vault rotates its own database credentials

# For static secrets, create rotation policy
vault write sys/rotate-root/config \
 rotation_period="720h" # 30 days

# Encrypt data without storing it
vault write transit/encrypt/orders \
 plaintext=$(echo "sensitive data" | base64)

vault write transit/decrypt/orders \
 ciphertext="vault:v1:..."

# After pod restart, must unseal
kubectl exec vault-0 -- vault status
# Sealed: true

# Must provide unseal keys
kubectl exec vault-0 -- vault operator unseal <key-1>
kubectl exec vault-0 -- vault operator unseal <key-2>
kubectl exec vault-0 -- vault operator unseal <key-3>

Open Source (self-hosted):
- Infrastructure: $200-500/month (HA cluster)
- Operations: 0.25 FTE = $3,000-5,000/month
Total: $3,200-5,500/month

Enterprise:
- License: $15,000-50,000/year
- Infrastructure: $200-500/month
- Operations: 0.25 FTE
Total: $4,500-9,500/month

Application → AWS SDK → Secrets Manager → KMS
 ↓
 (Encrypted storage)

# Create secret
aws secretsmanager create-secret \
 --name production/database/password \
 --description "Production database password" \
 --secret-string "supersecret123"

# Create with JSON structure
aws secretsmanager create-secret \
 --name production/database/config \
 --secret-string '{
 "username": "dbuser",
 "password": "supersecret123",
 "host": "db.example.com",
 "port": 5432
 }'

import boto3
import json

def get_secret():
 client = boto3.client('secretsmanager', region_name='us-east-1')

 response = client.get_secret_value(
 SecretId='production/database/config'
 )

 secret = json.loads(response['SecretString'])
 return secret

# Use in application
secret = get_secret()
db_url = f"postgresql://{secret['username']}:{secret['password']}@{secret['host']}:{secret['port']}/mydb"

# Install External Secrets Operator
helm install external-secrets \
 external-secrets/external-secrets \
 -n external-secrets-system \
 --create-namespace

# Create SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
 name: aws-secrets-manager
 namespace: production
spec:
 provider:
 aws:
 service: SecretsManager
 region: us-east-1
 auth:
 jwt:
 serviceAccountRef:
 name: external-secrets-sa

---
# Create ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: database-config
 namespace: production
spec:
 refreshInterval: 1h
 secretStoreRef:
 name: aws-secrets-manager
 kind: SecretStore
 target:
 name: database-config
 creationPolicy: Owner
 data:
 - secretKey: username
 remoteRef:
 key: production/database/config
 property: username
 - secretKey: password
 remoteRef:
 key: production/database/config
 property: password

# Lambda function for rotation
import boto3
import pymysql

def lambda_handler(event, context):
 service_client = boto3.client('secretsmanager')

 # Get current secret
 current_secret = service_client.get_secret_value(
 SecretId=event['SecretId']
 )

 # Generate new password
 new_password = generate_random_password()

 # Update database
 connection = pymysql.connect(
 host=current_secret['host'],
 user=current_secret['username'],
 password=current_secret['password']
 )

 with connection.cursor() as cursor:
 cursor.execute(
 f"ALTER USER '{current_secret['username']}' IDENTIFIED BY '{new_password}'"
 )
 connection.commit()

 # Update secret
 service_client.put_secret_value(
 SecretId=event['SecretId'],
 SecretString=json.dumps({
 'username': current_secret['username'],
 'password': new_password
 })
 )

# Enable automatic rotation
aws secretsmanager rotate-secret \
 --secret-id production/database/password \
 --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-secret \
 --rotation-rules AutomaticallyAfterDays=30

aws secretsmanager replicate-secret-to-regions \
 --secret-id production/database/password \
 --add-replica-regions Region=eu-west-1 \
 --add-replica-regions Region=ap-southeast-1

100 secrets, 1M API calls/month:

- Secrets: 100 × $0.40 = $40/month
- API calls: 1M × $0.05/10K = $5/month
Total: $45/month

1,000 secrets, 10M API calls/month:
- Secrets: 1,000 × $0.40 = $400/month
- API calls: 10M × $0.05/10K = $50/month
Total: $450/month

Developer → SOPS → KMS/PGP → Encrypted File → Git
 ↓
 (Encrypt/Decrypt)
 ↓
 Application

# Original secrets.yaml
api:
 key: sk_live_51H...
 secret: whsec_...
database:
 password: supersecret123
 host: db.example.com

# Encrypted with SOPS
api:
 key: ENC[AES256_GCM,data:abc123...,iv:xyz...,tag:def...]
 secret: ENC[AES256_GCM,data:uvw456...,iv:rst...,tag:ghi...]
database:
 password: ENC[AES256_GCM,data:mno789...,iv:jkl...,tag:pqr...]
 host: db.example.com # Not encrypted (no sensitive data)
sops:
 kms:
 - arn: arn:aws:kms:us-east-1:123456789:key/abc-123
 created_at: "2024-01-15T10:00:00Z"
 pgp:
 - fingerprint: ABC123...

# Install SOPS
brew install sops

# Or download binary
curl -LO https://github.com/mozilla/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64
chmod +x sops-v3.8.1.linux.amd64
sudo mv sops-v3.8.1.linux.amd64 /usr/local/bin/sops

creation_rules:
 # Production secrets (AWS KMS)
 - path_regex: production/.*\.yaml$
 kms: arn:aws:kms:us-east-1:123456789:key/production-key
 encrypted_regex: ^(data|stringData|password|secret|key)$

 # Staging secrets (different KMS key)
 - path_regex: staging/.*\.yaml$
 kms: arn:aws:kms:us-east-1:123456789:key/staging-key
 encrypted_regex: ^(data|stringData|password|secret|key)$

 # Development (PGP)
 - path_regex: development/.*\.yaml$
 pgp: >-
 ABC123DEF456,
 GHI789JKL012
 encrypted_regex: ^(data|stringData|password|secret|key)$

# Encrypt file
sops --encrypt secrets.yaml > secrets.enc.yaml

# Edit encrypted file (decrypts, opens editor, re-encrypts on save)
sops secrets.enc.yaml

# Decrypt file
sops --decrypt secrets.enc.yaml

# Decrypt and pipe to kubectl
sops --decrypt secrets.enc.yaml | kubectl apply -f -

# Flux Kustomization with SOPS decryption
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: apps
 namespace: flux-system
spec:
 interval: 10m
 path: ./apps/production
 prune: true
 sourceRef:
 kind: GitRepository
 name: flux-system
 # Decrypt SOPS-encrypted files
 decryption:
 provider: sops
 secretRef:
 name: sops-kms
---
# KMS credentials for decryption
apiVersion: v1
kind: Secret
metadata:
 name: sops-kms
 namespace: flux-system
type: Opaque
stringData:
 AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
 AWS_SECRET_ACCESS_KEY: wJalrXUtn...

# Install SOPS plugin
apiVersion: v1
kind: ConfigMap
metadata:
 name: argocd-cm
 namespace: argocd
data:
 kustomize.buildOptions: --enable-alpha-plugins --enable-helm
 configManagementPlugins: |
 - name: sops
 generate:
 command: ["sh", "-c"]
 args: ["sops -d secrets.yaml | kubectl apply -f -"]

SOPS:
- Software: Free
- KMS usage: ~$1/month (per key)
- Operations: Minimal
Total: ~$1-5/month

Vault:
- Encryption at rest ✓
- Encryption in transit ✓
- Detailed audit logs ✓
- Fine-grained access ✓
- Secret rotation ✓
- Dynamic secrets ✓
Score: 10/10

Secrets Manager:
- Encryption at rest ✓
- Encryption in transit ✓
- CloudTrail audit ✓
- IAM access control ✓
- Automatic rotation ✓ (limited)
- Dynamic secrets ✗
Score: 8/10

SOPS:
- Encryption at rest ✓
- Encryption in transit ⚠️ (Git over HTTPS)
- Git audit logs ⚠️
- KMS access control ✓
- Secret rotation ✗
- Dynamic secrets ✗
Score: 5/10

Vault: HIGH
- Setup HA cluster
- Configure storage backend
- Implement unsealing strategy
- Backup and recovery
- Monitoring and alerting
- Regular updates
Time: 40 hours initial + 20 hours/month

Secrets Manager: NONE
- Fully managed
- No infrastructure
- Automatic updates
Time: 2 hours initial + 1 hour/month

SOPS: LOW
- Install binary
- Configure .sops.yaml
- Manage KMS keys
Time: 4 hours initial + 2 hours/month

# SOPS-encrypted Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
 name: vault-config
type: Opaque
stringData:
 vault-token: ENC[AES256_GCM,data:abc123...]
 vault-addr: https://vault.example.com

# Application uses Vault for dynamic DB credentials
import hvac

# Vault token from SOPS-encrypted Kubernetes Secret
client = hvac.Client(
 url=os.getenv('VAULT_ADDR'),
 token=os.getenv('VAULT_TOKEN')
)

# Get dynamic database credentials
db_creds = client.secrets.database.generate_credentials(
 name='readonly'
)

# Sensitive: Secrets Manager
aws secretsmanager create-secret \
 --name production/database/password \
 --secret-string "supersecret"

# Non-sensitive: Parameter Store (cheaper)
aws ssm put-parameter \
 --name /production/database/host \
 --value "db.example.com" \
 --type String

# Vault: Minimal policy
path "secret/data/myapp/*" {
 capabilities = ["read"]
}

path "database/creds/readonly" {
 capabilities = ["read"]
}

# Deny everything else (implicit)

//AWSIAM:Minimalpolicy{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"secretsmanager:GetSecretValue","Resource":"arn:aws:secretsmanager:us-east-1:123456789:secret:production/myapp/*"}]}

Rotation schedule:
- API keys: 90 days
- Passwords: 30-60 days
- Certificates: 90 days (Let's Encrypt)
- Encryption keys: Yearly

# Vault: Enable audit logging
vault audit enable file file_path=/var/log/vault/audit.log

# AWS: Enable CloudTrail for Secrets Manager
aws cloudtrail create-trail \
 --name secrets-audit \
 --s3-bucket-name audit-logs

# Alert on secret access
aws cloudwatch put-metric-alarm \
 --alarm-name high-secret-access \
 --metric-name GetSecretValue \
 --threshold 1000 \
 --comparison-operator GreaterThanThreshold

# Bad
logger.info(f"Connecting with password: {password}")

# Good
logger.info("Connecting to database")
# Don't log secrets

Need dynamic secrets or automatic rotation?
├─ Yes → Vault or Secrets Manager
│ ├─ AWS-only?
│ │ ├─ Yes → Secrets Manager
│ │ └─ No → Vault
│ └─ Have ops team?
│ ├─ Yes → Vault
│ └─ No → Secrets Manager
└─ No → SOPS or Secrets Manager
 ├─ Using GitOps?
 │ ├─ Yes → SOPS
 │ └─ No → Secrets Manager
 └─ Budget?
 ├─ Limited → SOPS
 └─ Flexible → Secrets Manager