[{"data":1,"prerenderedAt":356},["ShallowReactive",2],{"blog-en-k3s-sealed-secrets-gitops-rebuild-private-key-lost":3,"blog-related-en-k3s-sealed-secrets-gitops-rebuild-private-key-lost":304,"blog-en-k3s-sealed-secrets-gitops-rebuild-private-key-lost-alt":292},{"id":4,"title":5,"author":6,"body":7,"date":286,"description":287,"extension":288,"image":289,"locale":290,"meta":291,"navigation":292,"path":293,"seo":294,"stem":295,"tags":296,"__hash__":303},"blog\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost.md","We Had It All as Code in GitOps. Then We Rebuilt the K3s Cluster, and Every Sealed Secret Became Undecryptable","Kubo Team",{"type":8,"value":9,"toc":276},"minimark",[10,15,23,26,34,45,49,55,67,88,95,99,105,120,129,133,139,149,157,161,167,170,220,233,242,249,253,256,263],[11,12,14],"h2",{"id":13},"_1-the-incident-argocd-shows-everything-synced-but-the-app-wont-even-start","1. The Incident — ArgoCD Shows Everything Synced, But the App Won't Even Start",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost\u002Fsection01.webp",[16,24,25],{},"Rebuilding a K3s cluster through GitOps is supposed to mean \"everything is code.\" Stand up the nodes with Terraform, point ArgoCD at the Git repository, and dozens of manifests fall into place as Synced one after another. Yet a handful of applications never become Ready. The logs show them crashing the moment they try to read a database password or an API token.",[16,27,28,29,33],{},"Digging further, the cause turns out to be in an unexpected place. ",[30,31,32],"strong",{},"At the moment the K3s cluster was rebuilt, every single Sealed Secret became undecryptable."," The encrypted SealedSecret resources were correctly committed to the Git repository, and ArgoCD's apply logs showed no errors. Yet none of them materialized as actual Secrets. The one piece that was supposedly \"code\" — the secrets — lost its reproducibility the moment the cluster was rebuilt.",[16,35,36,37,44],{},"This kind of trap — \"an operation that was supposedly code, but one piece of it can't be reproduced on rebuild\" — only grows more common the more a team builds and runs K3s clusters on its own. Whether you reach for a managed K3s environment like ",[38,39,43],"a",{"href":40,"rel":41},"https:\u002F\u002Fkubo.hexabase.io\u002F",[42],"nofollow","Kubo"," often comes down to exactly this kind of unglamorous operational design detail.",[11,46,48],{"id":47},"_2-the-cause-the-sealed-secrets-private-key-becomes-a-different-key-every-time-the-cluster-is-rebuilt","2. The Cause — The Sealed Secrets Private Key Becomes a Different Key Every Time the Cluster Is Rebuilt",[16,50,51],{},[19,52],{"alt":53,"src":54},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost\u002Fsection02.webp",[16,56,57,58,62,63,66],{},"Sealed Secrets works by using the ",[59,60,61],"code",{},"kubeseal"," CLI to encrypt a plaintext Secret with a public key, turning it into a ",[59,64,65],{},"SealedSecret"," custom resource that can safely be committed to Git. Only the controller running inside the cluster holds the matching private key; when a SealedSecret resource is applied, the controller decrypts it and generates a normal Kubernetes Secret.",[16,68,69,70,73,74,77,78,83,84,87],{},"What's easy to overlook is that ",[30,71,72],{},"the private key itself is never stored in Git",". The key exists only inside the cluster, as an ordinary Kubernetes Secret in the ",[59,75,76],{},"kube-system"," namespace. As the ",[38,79,82],{"href":80,"rel":81},"https:\u002F\u002Fraw.githubusercontent.com\u002Fbitnami-labs\u002Fsealed-secrets\u002Fmain\u002FREADME.md",[42],"official project documentation"," notes, this key never leaves the cluster unless you explicitly back it up with ",[59,85,86],{},"kubectl get secret -n kube-system -l sealedsecrets.bitnami.com\u002Fsealed-secrets-key",".",[16,89,90,91,94],{},"In this case, the operation was to build a brand-new K3s cluster from scratch and then pour the manifests in through GitOps. In that workflow, the newly installed Sealed Secrets controller ",[30,92,93],{},"automatically generates a completely new key pair"," on startup. The SealedSecrets already committed to Git can only be decrypted by the old cluster's key, so to a controller holding a different key, they're nothing but encrypted garbage. Even though GitOps had \"everything as code,\" the one thing that wasn't code — the key — failed to carry over.",[11,96,98],{"id":97},"_3-the-misconception-that-k3s-backs-up-etcd-so-were-fine","3. The Misconception That \"K3s Backs Up etcd, So We're Fine\"",[16,100,101],{},[19,102],{"alt":103,"src":104},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost\u002Fsection03.webp",[16,106,107,108,116,117,87],{},"It's worth clarifying how this relates to K3s's standard backup mechanism. When K3s uses the embedded etcd datastore, ",[38,109,112,113],{"href":110,"rel":111},"https:\u002F\u002Fdocs.k3s.io\u002Fdatastore\u002Fbackup-restore",[42],"periodic snapshots via ",[59,114,115],{},"k3s etcd-snapshot"," are enabled by default, taken at midnight and noon and kept for five generations. Because that snapshot includes every Secret, including the private key, ",[30,118,119],{},"restoring correctly from an etcd snapshot also restores the Sealed Secrets key, and nothing breaks",[16,121,122,123,128],{},"The failure only happens when, as in this case, the operation isn't \"restore from a snapshot\" but rather \"build a new cluster from scratch and pour manifests in via GitOps.\" If you don't recognize this distinction and simply assume \"we have backups, so we're fine,\" you won't find out otherwise until the moment you actually need to rebuild. It's also worth noting that K3s's documentation requires ",[38,124,127],{"href":125,"rel":126},"https:\u002F\u002Fdocs.k3s.io\u002Fcli\u002Fetcd-snapshot",[42],"backing up the server token file in addition to the datastore snapshot itself"," — an easily missed detail, since a mismatched token renders the snapshot unusable.",[11,130,132],{"id":131},"_4-recovery-whether-you-have-a-key-backup-changes-everything","4. Recovery — Whether You Have a Key Backup Changes Everything",[16,134,135],{},[19,136],{"alt":137,"src":138},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost\u002Fsection04.webp",[16,140,141,142,145,146,87],{},"If a key backup survives, recovery is straightforward. Apply the backed-up key YAML to the new cluster with ",[59,143,144],{},"kubectl apply",", restart the Sealed Secrets controller pod, and the controller reloads the existing key so it can correctly decrypt the old SealedSecrets sitting in Git. Even if the cluster itself is down, offline decryption is possible with ",[59,147,148],{},"kubeseal --recovery-unseal",[16,150,151,152,156],{},"If no key backup survived, the situation is harsh. As ",[38,153,155],{"href":80,"rel":154},[42],"the Sealed Secrets documentation"," states plainly, there is no way to decrypt a SealedSecret once its key is gone. In that case, for each affected secret, you have to manually re-enter the original plaintext value — from wherever it was kept outside Git, such as a password manager or a cloud provider's secret store — and re-seal it with the new key. In other words, how well you've tracked down \"where the original secret lives outside of Git\" determines how much time the incident costs you.",[11,158,160],{"id":159},"_5-prevention-turn-key-backup-and-rotation-into-a-standing-operational-practice","5. Prevention — Turn Key Backup and Rotation Into a Standing Operational Practice",[16,162,163],{},[19,164],{"alt":165,"src":166},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost\u002Fsection05.webp",[16,168,169],{},"There are three main ways to prevent a repeat.",[171,172,173,190,196],"ul",{},[174,175,176,179,180,183,184,189],"li",{},[30,177,178],{},"Make key backup a documented procedure",": Save the output of ",[59,181,182],{},"kubectl get secret -n kube-system -l sealedsecrets.bitnami.com\u002Fsealed-secrets-key -o yaml"," somewhere secure outside of Git — HashiCorp Vault, or an offline encrypted archive. ",[38,185,188],{"href":186,"rel":187},"https:\u002F\u002Fwww.kubernetes.io\u002Fdocs\u002Fconcepts\u002Fsecurity\u002Fsecrets-good-practices\u002F",[42],"Kubernetes' own good-practices documentation for Secrets"," recommends, alongside enabling encryption-at-rest for Secret data in etcd, treating this kind of key the same way you'd treat a root password",[174,191,192,195],{},[30,193,194],{},"Understand the rotation behavior",": The Sealed Secrets controller generates a new key pair every 30 days by default, but keeps the old keys around so existing SealedSecrets can still be decrypted. Make sure your team understands that key rotation and rotating the actual secret values (like passwords) are two separate things",[174,197,198,201,202,207,208,213,214,219],{},[30,199,200],{},"Reconsider the recovery path itself",": Rather than \"build from scratch and pour in via GitOps,\" consider aligning with ",[38,203,206],{"href":204,"rel":205},"https:\u002F\u002Fargo-cd.readthedocs.io\u002Fen\u002Fstable\u002Foperator-manual\u002Fsecret-management\u002F",[42],"ArgoCD's own recommendation"," for \"destination-cluster secret management,\" and move toward a setup where ",[38,209,212],{"href":210,"rel":211},"https:\u002F\u002Fexternal-secrets.io\u002Flatest\u002Fintroduction\u002Fgetting-started\u002F",[42],"External Secrets Operator"," syncs secrets from an external store like ",[38,215,218],{"href":216,"rel":217},"https:\u002F\u002Fdeveloper.hashicorp.com\u002Fvault\u002Ftutorials\u002Fkubernetes",[42],"HashiCorp Vault",". In that architecture, the secrets themselves live outside the cluster, so rebuilding the cluster no longer carries the risk of losing the controller's key at all",[16,221,222,223,228,229,232],{},"As ",[38,224,227],{"href":225,"rel":226},"https:\u002F\u002Fwww.redhat.com\u002Fen\u002Fblog\u002Fa-guide-to-secrets-management-with-gitops-and-kubernetes",[42],"Red Hat's guide"," points out, there are two broad families for secrets management in GitOps: encrypting secrets and storing them in Git, versus storing only references to an external store in Git. Whichever you choose, ",[30,230,231],{},"mapping out in advance exactly what does and doesn't carry over when you rebuild a cluster"," is the single most effective way to prevent an incident like this one.",[16,234,235,236,241],{},"This pattern — \"we thought we had backups, but the piece we actually needed turned out to be missing\" — came up before in ",[38,237,240],{"href":238,"rel":239},"https:\u002F\u002Fkubo.hexabase.io\u002Fblog\u002Fja\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc",[42],"a case where restoring a K3s PVC from Velero turned out to be empty",". There's always a gap worth checking between \"the mechanism exists\" and \"it actually restores,\" whether you're talking about GitOps or backups.",[16,243,244,245,248],{},"The more operational design you build and maintain yourself for a K3s cluster, the more these key-management and backup pitfalls accumulate. ",[38,246,43],{"href":40,"rel":247},[42]," comes with Prometheus + Grafana monitoring and automated cert-manager certificate renewal built in standard — using a managed K3s environment that already bakes in this kind of unglamorous-but-critical operational design is worth considering.",[11,250,252],{"id":251},"_6-summary","6. Summary",[16,254,255],{},"The reason every Sealed Secret failed after a GitOps rebuild of a K3s cluster was a blind spot in \"everything as code\": the private key itself is never managed in Git. Restoring from an etcd snapshot carries the key along with it, but building a cluster from scratch and pouring in GitOps manifests causes the controller to generate a brand-new key automatically.",[16,257,258,259,87],{},"The fix is clear: store key backups somewhere secure, understand how rotation behaves, and where possible, consider moving toward ",[38,260,262],{"href":204,"rel":261},[42],"the external secret store integration ArgoCD recommends",[16,264,265,266,269,270,275],{},"If designing and testing this kind of key-management operation on top of everything else feels like too much to carry on your own, handing the whole operational design over to a managed K3s environment like ",[38,267,43],{"href":40,"rel":268},[42]," is an option worth exploring. ",[38,271,274],{"href":272,"rel":273},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[42],"Get in touch"," to talk through the challenges you're running into with your current setup.",{"title":277,"searchDepth":278,"depth":278,"links":279},"",2,[280,281,282,283,284,285],{"id":13,"depth":278,"text":14},{"id":47,"depth":278,"text":48},{"id":97,"depth":278,"text":98},{"id":131,"depth":278,"text":132},{"id":159,"depth":278,"text":160},{"id":251,"depth":278,"text":252},"2026-10-15","Rebuilding a K3s cluster with GitOps left every Sealed Secret undecryptable. The real cause was how the private key was handled. Here's the failure mechanism, the recovery steps, and how to prevent it from happening again.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost",{"title":5,"description":287},"blog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost",[297,298,299,300,301,302],"k3s","kubernetes","sealed-secrets","secrets-management","gitops","disaster-recovery","LopMP6b776uHx3WtMMf5lGRGahtBgXtvTgGIUrd5XYc",[305,314,323,332,341,349],{"path":306,"title":307,"description":308,"date":309,"tags":310},"\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc","Every Night the Velero Backup Said \"Completed.\" Then We Restored It on K3s and the PVC Was Empty","K3s's default local-path-provisioner doesn't support CSI snapshots. Velero backups can keep reporting \"Completed\" while the volume data is never actually protected, leaving restored PVCs empty. Here's how to diagnose it and fix it with File System Backup and restore drills.","2026-10-13",[297,298,311,312,302,313],"velero","backup","csi",{"path":315,"title":316,"description":317,"date":318,"tags":319},"\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert","I ran helm upgrade directly on an app deployed by K3s's HelmChart CRD, and the next reconcile wiped it out","The difference between K3s and Kubernetes isn't just a feature list. The HelmChart CRD is an automation layer that quietly reverts manual helm upgrades and kubectl edits — here's how it works and how to avoid it.","2026-09-29",[297,298,320,321,301,322],"helm-controller","helmchart-crd","reconcile",{"path":324,"title":325,"description":326,"date":327,"tags":328},"\u002Fblog\u002Fen\u002Fk3s-etcd-snapshot-restore-token-cluster-reset-pitfalls","The Automated etcd Snapshots Were Running Every Day. Here's Why K3s Restore Still Failed in Production","K3s's etcd snapshots looked healthy every day, yet restore failed during a real outage. Three real-world traps — token management, S3 restore incidents, and cluster-reset pitfalls — reveal what's really at risk in embedded etcd backups.","2026-09-21",[297,298,329,330,302,331],"etcd","backup-restore","high-availability",{"path":333,"title":334,"description":335,"date":336,"tags":337},"\u002Fblog\u002Fen\u002Fcicd-pipeline-devsecops-gate-design-four-layers","Still Bolting Security Checks Onto Your CI\u002FCD Pipeline Without Deciding Where They Belong? A Four-Layer Model for DevSecOps Gate Design","You've adopted SAST, DAST, and container scanning — yet incidents still slip into production. This article reframes CI\u002FCD pipeline security as a four-layer design problem and walks through concrete implementation patterns for K3s\u002FKubernetes environments.","2026-08-28",[297,298,338,339,340,301],"ci-cd","devsecops","admission-control",{"path":342,"title":343,"description":344,"date":345,"tags":346},"\u002Fblog\u002Fen\u002Fkubernetes-gitops-branch-antipattern-fleet-scaling","Your dev\u002Fstaging\u002Fprod Branches Are a Time Bomb: Why Kubernetes GitOps Really Breaks","Splitting dev\u002Fstaging\u002Fproduction by Git branch is a GitOps anti-pattern that undermines Kubernetes' declarative foundations. Learn why drift happens, how to migrate to a directory-based, trunk-based setup, and how to design for fleet-scale growth.","2026-08-10",[297,298,301,347,348],"argocd","devops",{"path":350,"title":351,"description":352,"date":353,"tags":354},"\u002Fblog\u002Fen\u002Fkubernetes-image-signing-sigstore-supply-chain","Anyone Can Rewrite an Image Tag. Why Kubernetes Needs Sigstore-Backed Signing to Prove Provenance","Container image signing explained: tags can be overwritten by anyone, and passing CI tests doesn't guarantee the image running in production is the one you built. Learn how Sigstore and Kyverno work together to reject unsigned images on Kubernetes\u002FK3s, integrated into a GitOps workflow.","2026-08-06",[297,298,338,301,355],"security",1792592639522]