1. The Incident — ArgoCD Shows Everything Synced, But the App Won't Even Start
Rebuilding a K3s cluster through GitOps is supposed to mean "everything is code." Stand up the nodes with Terraform, point ArgoCD at the Git repository, and dozens of manifests fall into place as Synced one after another. Yet a handful of applications never become Ready. The logs show them crashing the moment they try to read a database password or an API token.
Digging further, the cause turns out to be in an unexpected place. At the moment the K3s cluster was rebuilt, every single Sealed Secret became undecryptable. The encrypted SealedSecret resources were correctly committed to the Git repository, and ArgoCD's apply logs showed no errors. Yet none of them materialized as actual Secrets. The one piece that was supposedly "code" — the secrets — lost its reproducibility the moment the cluster was rebuilt.
This kind of trap — "an operation that was supposedly code, but one piece of it can't be reproduced on rebuild" — only grows more common the more a team builds and runs K3s clusters on its own. Whether you reach for a managed K3s environment like Kubo often comes down to exactly this kind of unglamorous operational design detail.
2. The Cause — The Sealed Secrets Private Key Becomes a Different Key Every Time the Cluster Is Rebuilt
Sealed Secrets works by using the kubeseal CLI to encrypt a plaintext Secret with a public key, turning it into a SealedSecret custom resource that can safely be committed to Git. Only the controller running inside the cluster holds the matching private key; when a SealedSecret resource is applied, the controller decrypts it and generates a normal Kubernetes Secret.
What's easy to overlook is that the private key itself is never stored in Git. The key exists only inside the cluster, as an ordinary Kubernetes Secret in the kube-system namespace. As the official project documentation notes, this key never leaves the cluster unless you explicitly back it up with kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key.
In this case, the operation was to build a brand-new K3s cluster from scratch and then pour the manifests in through GitOps. In that workflow, the newly installed Sealed Secrets controller automatically generates a completely new key pair on startup. The SealedSecrets already committed to Git can only be decrypted by the old cluster's key, so to a controller holding a different key, they're nothing but encrypted garbage. Even though GitOps had "everything as code," the one thing that wasn't code — the key — failed to carry over.
3. The Misconception That "K3s Backs Up etcd, So We're Fine"
It's worth clarifying how this relates to K3s's standard backup mechanism. When K3s uses the embedded etcd datastore, periodic snapshots via k3s etcd-snapshot are enabled by default, taken at midnight and noon and kept for five generations. Because that snapshot includes every Secret, including the private key, restoring correctly from an etcd snapshot also restores the Sealed Secrets key, and nothing breaks.
The failure only happens when, as in this case, the operation isn't "restore from a snapshot" but rather "build a new cluster from scratch and pour manifests in via GitOps." If you don't recognize this distinction and simply assume "we have backups, so we're fine," you won't find out otherwise until the moment you actually need to rebuild. It's also worth noting that K3s's documentation requires backing up the server token file in addition to the datastore snapshot itself — an easily missed detail, since a mismatched token renders the snapshot unusable.
4. Recovery — Whether You Have a Key Backup Changes Everything
If a key backup survives, recovery is straightforward. Apply the backed-up key YAML to the new cluster with kubectl apply, restart the Sealed Secrets controller pod, and the controller reloads the existing key so it can correctly decrypt the old SealedSecrets sitting in Git. Even if the cluster itself is down, offline decryption is possible with kubeseal --recovery-unseal.
If no key backup survived, the situation is harsh. As the Sealed Secrets documentation states plainly, there is no way to decrypt a SealedSecret once its key is gone. In that case, for each affected secret, you have to manually re-enter the original plaintext value — from wherever it was kept outside Git, such as a password manager or a cloud provider's secret store — and re-seal it with the new key. In other words, how well you've tracked down "where the original secret lives outside of Git" determines how much time the incident costs you.
5. Prevention — Turn Key Backup and Rotation Into a Standing Operational Practice
There are three main ways to prevent a repeat.
- Make key backup a documented procedure: Save the output of
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yamlsomewhere secure outside of Git — HashiCorp Vault, or an offline encrypted archive. Kubernetes' own good-practices documentation for Secrets recommends, alongside enabling encryption-at-rest for Secret data in etcd, treating this kind of key the same way you'd treat a root password - Understand the rotation behavior: The Sealed Secrets controller generates a new key pair every 30 days by default, but keeps the old keys around so existing SealedSecrets can still be decrypted. Make sure your team understands that key rotation and rotating the actual secret values (like passwords) are two separate things
- Reconsider the recovery path itself: Rather than "build from scratch and pour in via GitOps," consider aligning with ArgoCD's own recommendation for "destination-cluster secret management," and move toward a setup where External Secrets Operator syncs secrets from an external store like HashiCorp Vault. In that architecture, the secrets themselves live outside the cluster, so rebuilding the cluster no longer carries the risk of losing the controller's key at all
As Red Hat's guide points out, there are two broad families for secrets management in GitOps: encrypting secrets and storing them in Git, versus storing only references to an external store in Git. Whichever you choose, mapping out in advance exactly what does and doesn't carry over when you rebuild a cluster is the single most effective way to prevent an incident like this one.
This pattern — "we thought we had backups, but the piece we actually needed turned out to be missing" — came up before in a case where restoring a K3s PVC from Velero turned out to be empty. There's always a gap worth checking between "the mechanism exists" and "it actually restores," whether you're talking about GitOps or backups.
The more operational design you build and maintain yourself for a K3s cluster, the more these key-management and backup pitfalls accumulate. Kubo comes with Prometheus + Grafana monitoring and automated cert-manager certificate renewal built in standard — using a managed K3s environment that already bakes in this kind of unglamorous-but-critical operational design is worth considering.
6. Summary
The reason every Sealed Secret failed after a GitOps rebuild of a K3s cluster was a blind spot in "everything as code": the private key itself is never managed in Git. Restoring from an etcd snapshot carries the key along with it, but building a cluster from scratch and pouring in GitOps manifests causes the controller to generate a brand-new key automatically.
The fix is clear: store key backups somewhere secure, understand how rotation behaves, and where possible, consider moving toward the external secret store integration ArgoCD recommends.
If designing and testing this kind of key-management operation on top of everything else feels like too much to carry on your own, handing the whole operational design over to a managed K3s environment like Kubo is an option worth exploring. Get in touch to talk through the challenges you're running into with your current setup.