Security

Managing Secrets at Scale

● Advanced ⏱ 15 min read

Kubernetes Secrets are not secret by default. They are base64-encoded, stored in etcd, and readable by anyone with get secrets permission in that namespace. At small scale that is manageable. At scale — dozens of teams, hundreds of namespaces, credentials rotated by external systems — you need a deliberate strategy. This guide covers the spectrum from encryption at rest to full external secret stores.

The Base64 Problem

Base64 is encoding, not encryption. Anyone who can read a Secret manifest from etcd or via kubectl get secret -o yaml gets the plaintext value immediately:

# This is NOT secure storage — it's just encoding
echo "bXlwYXNzd29yZA==" | base64 -d
# mypassword

Three real risks:

etcd compromise — without encryption at rest, anyone with etcd access reads all secrets.
git exposure — a Secret YAML committed to a repo exposes the value immediately.
RBAC oversharing — list secrets returns all values, not just metadata. A single wide binding leaks everything.

Encryption at Rest

The API server can encrypt Secret data before writing it to etcd. Configure an EncryptionConfiguration and pass it to kube-apiserver via --encryption-provider-config.

EncryptionConfiguration — AES-GCM

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets", "configmaps"]
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>   # generate: head -c 32 /dev/urandom | base64
  - identity: {}    # fallback — allows reading unencrypted existing secrets during migration

After enabling, existing secrets are not automatically re-encrypted. Re-encrypt all secrets in one pass:

# Re-encrypt all secrets by rewriting them through the API server
kubectl get secrets -A -o json | kubectl replace -f -

For managed clusters (EKS, GKE, AKS), encryption at rest is usually enabled per-cluster through the cloud provider's key management (AWS KMS, Cloud KMS, Azure Key Vault). Prefer that over managing your own key material.

External Secrets Operator

External Secrets Operator (ESO) syncs secrets from external stores (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault, 1Password, and more) into Kubernetes Secrets. Pods consume standard Kubernetes Secrets — the operator handles the sync and refresh.

Secret delivery architectures — native vs. external store

Native K8s Secrets

CI/CD pipeline

↓

kubectl apply secret.yaml

↓

etcd (base64 / encrypted)

↓

Pod env / volume mount

⚠️ value lives in etcd & git

External Secrets Operator

Vault / AWS SM / GCP SM

↓

ExternalSecret CRD

↓

ESO controller syncs

↓

K8s Secret (auto-refreshed)

↓

Pod env / volume mount

✅ source of truth stays in vault

Native Secrets require CI/CD to push values into the cluster. ESO pulls from the external store and syncs into K8s Secrets automatically — the plaintext never lives in git.

SecretStore — AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore             # ClusterSecretStore for cross-namespace access
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:                  # IRSA — use pod's ServiceAccount JWT for auth
          serviceAccountRef:
            name: eso-sa

ExternalSecret — sync a specific secret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h          # re-sync every hour
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-credentials       # K8s Secret name to create/update
    creationPolicy: Owner      # ESO owns and manages this Secret
    deletionPolicy: Retain     # keep K8s Secret if ExternalSecret is deleted
  data:
  - secretKey: DB_PASSWORD     # key in the K8s Secret
    remoteRef:
      key: prod/myapp/db       # path in AWS Secrets Manager
      property: password       # JSON key inside the secret value

Sealed Secrets

Sealed Secrets (by Bitnami/VMware) takes the opposite approach: encrypt the Secret client-side so the encrypted form is safe to commit to git. Only the cluster's sealed-secrets-controller holds the private key to decrypt it.

kubeseal — encrypt a Secret for git

# Install kubeseal CLI, then encrypt
kubectl create secret generic db-password \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal \
    --controller-name sealed-secrets-controller \
    --controller-namespace kube-system \
    --format yaml \
  > sealed-db-password.yaml   # safe to commit to git

# Apply to cluster — controller decrypts and creates the real Secret
kubectl apply -f sealed-db-password.yaml

Sealed Secrets are scoped by namespace and name — a SealedSecret for production/db-password cannot be decrypted in staging. This scoping is baked into the encryption.

Secrets Store CSI Driver

The Secrets Store CSI Driver mounts secrets directly from external stores as files in the pod — bypassing etcd entirely. The secret value never touches Kubernetes Secret objects.

SecretProviderClass — mount from Vault

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: production
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "myapp"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/prod/myapp"
        secretKey: "password"

pod — mount via CSI driver

spec:
  containers:
  - name: app
    image: myapp:1.0.0
    volumeMounts:
    - name: secrets
      mountPath: "/mnt/secrets"   # secret appears as file at /mnt/secrets/db-password
      readOnly: true
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-db-creds"

Rotation Patterns

Secret rotation is where most approaches break down in practice. The key questions are:

How does the new value reach the pod? — File mounts update automatically when ESO or CSI driver refreshes; env vars require a pod restart.
Does the app handle both old and new values during rotation? — Databases typically accept both passwords during a rotation window.
Who triggers the rotation? — External systems (AWS Secrets Manager rotation) vs. manual vs. scheduled job.

💡

env vars vs. file mounts for rotation

Env vars are set at pod start and do not update when the underlying Secret changes. File-mounted secrets (via volumeMount) update when the kubelet syncs — default sync period is 60 seconds. For secrets that rotate, prefer file mounts over env vars.

What to Avoid

Never commit plaintext Secrets to git — even private repos. Use Sealed Secrets or reference external stores.
Never list or watch secrets cluster-wide from application code. Scope RBAC to get with specific resourceNames.
Don't use stringData in manifests stored anywhere — it's plaintext in YAML form.
Avoid long-lived static tokens — prefer IRSA/Workload Identity where the cloud provider generates short-lived tokens per pod automatically.
Don't log env vars — env printed to stdout during debugging exposes all pod environment variables including secrets.

kubectl Commands

# Create a Secret imperatively (never commit the resulting YAML)
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=supersecret \
  -n production

# Decode a specific key
kubectl get secret db-creds -n production \
  -o jsonpath='{.data.password}' | base64 -d

# List all secrets in a namespace (metadata only — doesn't decode)
kubectl get secrets -n production

# Check which pods mount a specific Secret
kubectl get pods -n production -o json | \
  jq '.items[] | select(
    .spec.volumes[]? | .secret?.secretName == "db-creds"
  ) | .metadata.name'

# Force ESO to re-sync immediately
kubectl annotate externalsecret db-credentials \
  force-sync=$(date +%s) --overwrite -n production