Secrets in GitOps: SOPS vs Sealed Secrets vs External Secrets Operator

The first week of any GitOps migration includes the same conversation: "everything is declared in Git, so where do the database passwords go?" Committing plain Kubernetes Secrets is not an option, since base64 is not encryption. There are two families of answers: put encrypted secrets in Git, or put references in Git and keep the values in a secret manager.

We have run both in production across several clusters. This guide compares the main tools, explains the trade-offs that only show up after a few months, and gives a recommendation per situation.

Family 1: encrypted secrets in Git

SOPS

SOPS encrypts the values in a YAML or JSON file while leaving keys readable, so diffs show which keys changed. Encryption keys come from a KMS (AWS KMS, GCP KMS, Azure Key Vault), age, or PGP. Argo CD and Flux decrypt during sync, through a plugin for Argo CD or natively in Flux's kustomize controller.

# secrets/db.enc.yaml (committed)
apiVersion: v1
kind: Secret
metadata:
  name: db
stringData:
  password: ENC[AES256_GCM,data:3s9X...,iv:...,tag:...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:eu-west-1:111122223333:key/abcd-...

Sealed Secrets

A controller in the cluster holds a private key. Developers encrypt secrets with the cluster's public key using kubeseal and commit the resulting SealedSecret. Only that controller can decrypt it into a normal Secret.

kubectl create secret generic db --from-literal=password='s3cr3t' --dry-run=client -o yaml \
  | kubeseal --format yaml > sealed-db.yaml

What this family is good at

• Git is the full source of truth, including secret versions and history.
• No runtime dependency on an external service: the cluster can start with nothing but Git and its key.
• Simple mental model for small teams.

Where it hurts

• Rotation means a commit. Rotating a database password requires generating it, encrypting it, committing it and syncing, often across several clusters.
• Old ciphertexts live forever in Git history. If an encryption key is ever compromised, every historical secret encrypted with it is exposed. Key rotation does not re-encrypt history.
• Sealed Secrets are bound to one cluster's key (by default). Multi-cluster setups need a shared key or one sealed copy per cluster.
• Losing the controller key (Sealed Secrets) means losing the ability to decrypt everything; it must be backed up carefully.

Family 2: references in Git, values in a secret manager

External Secrets Operator (ESO)

ESO runs in the cluster and syncs values from AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault and others into Kubernetes Secrets. Git holds only the reference:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    name: db
  data:
    - secretKey: password
      remoteRef:
        key: prod/checkout/db
        property: password

Secrets Store CSI Driver

Mounts secrets from a provider directly into pods as files, optionally syncing them into Kubernetes Secrets too. Secrets never need to exist as Kubernetes objects if you only mount them.

What this family is good at

• Rotation without commits. Rotate in the secret manager (often automatically), and ESO picks up the new value at the next refresh.
• One source for many consumers: Kubernetes, serverless functions, CI and VMs read the same secret.
• Audit and access control come from the cloud provider: who read which secret, when.
• Nothing sensitive in Git history, even encrypted.

Where it hurts

• Runtime dependency. If the secret manager or the cluster's identity to it fails, new pods may not get secrets.
• Bootstrap. The cluster needs an identity to talk to the secret manager. Use workload identity (IRSA, EKS Pod Identity, GKE Workload Identity) rather than a static credential, or you have moved the problem rather than solved it.
• Drift is possible: the value in the manager can change without any Git commit. That is the point, and it also means Git alone does not tell you what is running.
• Rotation still needs a restart if apps read secrets from environment variables at startup (see our post on ConfigMap updates not reaching pods; the same applies to Secrets).

Side by side

Our recommendation

• Small team, one or two clusters, no cloud secret manager: Sealed Secrets or SOPS with age. Back up keys, and plan rotations as commits.
• Cloud-native, several clusters or non-Kubernetes consumers: External Secrets Operator with workload identity. This is what we standardised on.
• Strict requirement that secrets never exist as Kubernetes Secrets: CSI driver with mount-only.

Whatever you pick, a few rules apply everywhere: enable encryption at rest for Kubernetes Secrets in etcd, restrict get/list on Secrets with RBAC (listing secrets reveals their values), and never print secret values in CI logs or Argo CD diffs. Argo CD masks Secret data in its UI, but plugins and custom tools may not.

Common mistakes

Key takeaways

• Encrypted-in-Git is simpler; references-to-a-manager scales better and rotates easier.
• Encrypted history is permanent; plan for key compromise.
• Use workload identity, not static credentials, for secret managers.
• Protect Secrets in the cluster with RBAC and encryption at rest regardless of tool.
• Make rotation reach running processes.

References

• SOPS — https://github.com/getsops/sops
• Sealed Secrets — https://github.com/bitnami-labs/sealed-secrets
• External Secrets Operator — https://external-secrets.io/latest/
• Secrets Store CSI Driver — https://secrets-store-csi-driver.sigs.k8s.io/
• Kubernetes good practices for Secrets — https://kubernetes.io/docs/concepts/security/secrets-good-practices/