[{"data":1,"prerenderedAt":410},["ShallowReactive",2],{"blog-en-k3s-helmchart-crd-manual-helm-upgrade-reconcile-revert":3,"blog-related-en-k3s-helmchart-crd-manual-helm-upgrade-reconcile-revert":359,"blog-en-k3s-helmchart-crd-manual-helm-upgrade-reconcile-revert-alt":347},{"id":4,"title":5,"author":6,"body":7,"date":341,"description":342,"extension":343,"image":344,"locale":345,"meta":346,"navigation":347,"path":348,"seo":349,"stem":350,"tags":351,"__hash__":358},"blog\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert.md","I ran helm upgrade directly on an app deployed by K3s's HelmChart CRD, and the next reconcile wiped it out","Kubo Team",{"type":8,"value":9,"toc":330},"minimark",[10,15,24,31,42,53,57,68,91,97,114,118,136,149,155,171,177,181,190,214,218,221,260,266,275,279,286,292,304,308,320],[11,12,14],"h2",{"id":13},"_1-i-ran-helm-upgrade-directly-in-production-and-by-the-next-morning-the-config-was-gone","1. I ran helm upgrade directly in production, and by the next morning the config was gone",[16,17,18,19,23],"p",{},"An application deployed via K3s's HelmChart CRD needed an urgent parameter change. I ran ",[20,21,22],"code",{},"helm upgrade --set"," to overwrite the value directly, confirmed the Pod came up with the correct configuration, and moved on.",[16,25,26],{},[27,28],"img",{"alt":29,"src":30},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert\u002Fsection01.webp",[16,32,33,34,37,38],{},"The next morning, after a routine node maintenance restart, the config had reverted to its state from before the upgrade. ",[20,35,36],{},"helm history"," showed the revision had incremented, but the Pod actually running was still on the old configuration. Digging into the difference between K3s and Kubernetes, I found the cause wasn't my mistake — it was ",[39,40,41],"strong",{},"an automation layer unique to K3s called the \"HelmChart CRD.\"",[16,43,44,45,52],{},"This kind of anxiety — not knowing what your cluster is automating behind your back — is something worth considering when choosing a K3s-based managed service, too. Services like ",[46,47,51],"a",{"href":48,"rel":49},"https:\u002F\u002Fkubo.hexabase.io\u002F",[50],"nofollow","Kubo"," emphasize operational transparency, but first you need to understand what K3s itself is doing.",[11,54,56],{"id":55},"_2-the-3-layer-structure-unique-to-k3s-helmchart-crd-helm-controller-kubernetes-job","2. The 3-layer structure unique to K3s: HelmChart CRD → helm-controller → Kubernetes Job",[16,58,59,60,63,64,67],{},"In vanilla Kubernetes, running ",[20,61,62],{},"helm install"," has the Helm client render the chart and create resources directly against the Kubernetes API. There is no \"declarative watcher\" sitting on top — nothing changes again until someone runs ",[20,65,66],{},"helm upgrade",".",[16,69,70,71,74,75,80,81,84,85,90],{},"K3s bundles something extra: its own ",[39,72,73],{},"Helm Controller",". According to ",[46,76,79],{"href":77,"rel":78},"https:\u002F\u002Fdocs.k3s.io\u002Fadd-ons\u002Fhelm",[50],"K3s's official documentation",", K3s uses a Custom Resource Definition (CRD) called ",[20,82,83],{},"HelmChart"," to declaratively manage the install, upgrade, and uninstall of Helm charts. This isn't limited to bundled components like Traefik or ServiceLB — ",[46,86,89],{"href":87,"rel":88},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-20-k3s-auto-deploying-helm-charts\u002Fview",[50],"you can use the same mechanism to auto-deploy your own custom Helm charts"," as well.",[16,92,93],{},[27,94],{"alt":95,"src":96},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert\u002Fsection02.webp",[16,98,99,100,105,106,109,110,113],{},"In other words, K3s always has a background structure running: \"HelmChart CRD (declaration) → helm-controller (watcher) → Kubernetes Job (execution).\" The ",[46,101,104],{"href":102,"rel":103},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fhelm-controller\u002Fblob\u002Fmaster\u002Fdoc\u002Fhelmchart.md",[50],"k3s-io\u002Fhelm-controller field reference"," documents many settings such as ",[20,107,108],{},"chartContent"," and ",[20,111,112],{},"valuesContent",", but it does not specify how changes made outside the CRD are handled. That gap is exactly where the trouble starts.",[11,115,117],{"id":116},"_3-what-actually-happened-failurepolicy-and-the-job-re-trigger","3. What actually happened: failurePolicy and the Job re-trigger",[16,119,120,123,124,127,128,131,132,135],{},[46,121,79],{"href":77,"rel":122},[50]," defines a ",[20,125,126],{},"failurePolicy"," setting whose default is ",[20,129,130],{},"\"reinstall\"",". When a Job is judged to have failed, this automatically performs a clean uninstall followed by a reinstall. Unless you switch it to ",[20,133,134],{},"\"abort\"",", helm-controller re-triggers a Helm-driven Job every time it detects a difference between the desired state (the HelmChart CRD's contents) and the actual state (what's really running in the cluster).",[16,137,138,139,144,145,148],{},"This behavior has been reported as a real-world problem multiple times. ",[46,140,143],{"href":141,"rel":142},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fhelm-controller\u002Fissues\u002F50",[50],"helm-controller Issue #50"," describes the controller repeatedly re-triggering the install Job and generating new revisions even though the chart version hadn't changed. The logs showed ",[20,146,147],{},"DesiredSet - Replace Wait batch\u002Fv1, Kind=Job",", indicating the controller kept detecting a need to replace the Job resource.",[16,150,151],{},[27,152],{"alt":153,"src":154},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert\u002Fsection03.webp",[16,156,157,162,163,166,167,170],{},[46,158,161],{"href":159,"rel":160},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F2893",[50],"k3s-io\u002Fk3s Issue #2893"," documents a similar case: a user had customized Traefik via ",[20,164,165],{},"HelmChartConfig"," (enabling the dashboard, adding TLS settings), and reported that \"every time K3s or the host restarts, this customization isn't picked up and the default Traefik settings apply instead.\" The workaround mentioned was forcing a re-read by adding whitespace or a newline to the manifest file. From the perspective of this 3-layer structure, a manual ",[20,168,169],{},"kubectl edit"," is nothing more than \"a diff that only survives temporarily.\"",[16,172,173,174],{},"Whether you can actually trace this Job re-trigger from a management console matters a lot for day-to-day operations. Kubo's Captain UI is designed to surface the state of Helm charts and Jobs running in the background, making it easier to notice when a reconcile has quietly reverted your configuration — reducing the \"wait, when did this change back?\" moments. ",[46,175,51],{"href":48,"rel":176},[50],[11,178,180],{"id":179},"_4-pitfall-2-some-resources-dont-roll-back-even-after-deleting-helmchartconfig","4. Pitfall #2: some resources don't roll back even after deleting HelmChartConfig",[16,182,183,184,189],{},"There's another wrinkle: reconcile behavior isn't consistent across every resource type. ",[46,185,188],{"href":186,"rel":187},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F5534",[50],"k3s-io\u002Fk3s Issue #5534"," reports that after customizing Traefik's ports via HelmChartConfig (to 1234\u002F5555) and then deleting that HelmChartConfig, the LoadBalancer-type Service and its klipper-lb Pod stayed on the custom values instead of reverting to defaults.",[16,191,192,193,109,198,203,204,207,208,213],{},"In other words, \"removing the HelmChartConfig always restores the defaults\" isn't a safe assumption either. Both the ",[46,194,197],{"href":195,"rel":196},"https:\u002F\u002Fforums.suse.com\u002Ft\u002Fk3s-traefik-configuration-override-problems\u002F37264",[50],"SUSE community forum",[46,199,202],{"href":200,"rel":201},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fdiscussions\u002F11176",[50],"k3s-io\u002Fk3s Discussion #11176"," show recurring reports of HelmChartConfig either not applying as expected or not reverting properly. K3s's Helm Controller watches ",[20,205,206],{},"\u002Fvar\u002Flib\u002Francher\u002Fk3s\u002Fserver\u002Fmanifests\u002F"," at startup and generates Jobs from what it finds there — and the sibling project ",[46,209,212],{"href":210,"rel":211},"https:\u002F\u002Fdocs.rke2.io\u002Fadd-ons\u002Fhelm",[50],"RKE2's official documentation"," describes the same mechanism, noting that manifests placed in that directory are auto-deployed as AddOn custom resources. Even so, the diff-detection logic isn't always intuitive on a per-resource-type basis.",[11,215,217],{"id":216},"_5-the-right-way-to-fix-it-change-through-helmchartconfig-and-failurepolicy-abort","5. The right way to fix it: change through HelmChartConfig, and failurePolicy: abort",[16,219,220],{},"Given this structure, the rules for changing an app or addon managed by the HelmChart CRD on K3s become clear.",[222,223,224,236,244,254],"ul",{},[225,226,227,235],"li",{},[39,228,229,230,232,233,67],{},"Don't change things directly with ",[20,231,169],{}," or ",[20,234,22],{}," Assume it will be wiped out on the next reconcile.",[225,237,238,243],{},[39,239,240,241,67],{},"Always make changes through HelmChartConfig's ",[20,242,112],{}," helm-controller recognizes this as a legitimate input, so it survives reconciliation.",[225,245,246,253],{},[39,247,248,249,252],{},"Set ",[20,250,251],{},"failurePolicy: abort"," if you want to avoid unintended automatic reinstalls."," On failure, this leaves the chart in a failed state awaiting manual intervention instead of silently reinstalling.",[225,255,256,259],{},[39,257,258],{},"When deleting a HelmChartConfig, verify each affected resource actually reverts to its default"," — as Issue #5534 shows, some resource types don't.",[16,261,262],{},[27,263],{"alt":264,"src":265},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert\u002Fsection04.webp",[16,267,268,269,274],{},"This is exactly the kind of behavior you'd miss if you think of K3s only as \"lightweight Kubernetes.\" ",[46,270,273],{"href":271,"rel":272},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fhelm-controller\u002Fissues\u002F351",[50],"helm-controller Issue #351"," shows the reconcile logic itself is still an active area of discussion and improvement — this is a structure worth staying alert to going forward.",[11,276,278],{"id":277},"_6-how-kubo-changes-this","6. How Kubo changes this",[16,280,281,282,285],{},"Even on K3s-based ",[46,283,51],{"href":48,"rel":284},[50],", the HelmChart CRD mechanism itself exists as the same standard K3s feature. The difference is whether this \"invisible automation layer\" stays a black box or becomes something you can operate with visibility.",[16,287,288,289,67],{},"Kubo's Captain UI is designed so you can check the state of Helm charts and Jobs running automatically inside the cluster from a management dashboard, making it easier to notice \"the reconcile ran and quietly reverted my config\" before it becomes a production incident. Automation you'd never see on EKS or AKS becomes something you can operate without a black box on ",[46,290,51],{"href":48,"rel":291},[50],[16,293,294,295,297,298,303],{},"And for finance, healthcare, and manufacturing organizations in air-gapped environments — where even manually editing ",[20,296,206],{}," is difficult — ",[46,299,302],{"href":300,"rel":301},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[50],"Kubo On-Premise"," offers an option built around \"Your Infrastructure. Your Rules. Your Kubernetes.\"",[11,305,307],{"id":306},"_7-summary","7. Summary",[16,309,310,311,314,315,232,317,319],{},"The difference between K3s and Kubernetes isn't limited to how ReplicaSets or networking are implemented. The ",[39,312,313],{},"HelmChart CRD + helm-controller + Kubernetes Job"," structure bundled with K3s is a unique layer that doesn't exist in vanilla Kubernetes — and if you don't know about it, a direct ",[20,316,169],{},[20,318,66],{}," will quietly get reverted on the next reconcile.",[16,321,322,323,325,326,329],{},"Always make changes through HelmChartConfig, and operate with a clear understanding of how ",[20,324,126],{}," behaves. That's the first step toward running a K3s cluster reliably in production. If you'd rather operate this automation layer with full visibility, ",[46,327,51],{"href":48,"rel":328},[50],"'s Captain UI is worth a look.",{"title":331,"searchDepth":332,"depth":332,"links":333},"",2,[334,335,336,337,338,339,340],{"id":13,"depth":332,"text":14},{"id":55,"depth":332,"text":56},{"id":116,"depth":332,"text":117},{"id":179,"depth":332,"text":180},{"id":216,"depth":332,"text":217},{"id":277,"depth":332,"text":278},{"id":306,"depth":332,"text":307},"2026-09-29","The difference between K3s and Kubernetes isn't just a feature list. The HelmChart CRD is an automation layer that quietly reverts manual helm upgrades and kubectl edits — here's how it works and how to avoid it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert",{"title":5,"description":342},"blog\u002Fen\u002Fk3s-helmchart-crd-manual-helm-upgrade-reconcile-revert",[352,353,354,355,356,357],"k3s","kubernetes","helm-controller","helmchart-crd","gitops","reconcile","4m6Q6DqReV4uFLXETAheAk3JJdTwS8B8izQ2h1ij3ho",[360,369,378,386,393,401],{"path":361,"title":362,"description":363,"date":364,"tags":365},"\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost","We Had It All as Code in GitOps. Then We Rebuilt the K3s Cluster, and Every Sealed Secret Became Undecryptable","Rebuilding a K3s cluster with GitOps left every Sealed Secret undecryptable. The real cause was how the private key was handled. Here's the failure mechanism, the recovery steps, and how to prevent it from happening again.","2026-10-15",[352,353,366,367,356,368],"sealed-secrets","secrets-management","disaster-recovery",{"path":370,"title":371,"description":372,"date":373,"tags":374},"\u002Fblog\u002Fen\u002Fcicd-pipeline-devsecops-gate-design-four-layers","Still Bolting Security Checks Onto Your CI\u002FCD Pipeline Without Deciding Where They Belong? A Four-Layer Model for DevSecOps Gate Design","You've adopted SAST, DAST, and container scanning — yet incidents still slip into production. This article reframes CI\u002FCD pipeline security as a four-layer design problem and walks through concrete implementation patterns for K3s\u002FKubernetes environments.","2026-08-28",[352,353,375,376,377,356],"ci-cd","devsecops","admission-control",{"path":379,"title":380,"description":381,"date":382,"tags":383},"\u002Fblog\u002Fen\u002Fkubernetes-gitops-branch-antipattern-fleet-scaling","Your dev\u002Fstaging\u002Fprod Branches Are a Time Bomb: Why Kubernetes GitOps Really Breaks","Splitting dev\u002Fstaging\u002Fproduction by Git branch is a GitOps anti-pattern that undermines Kubernetes' declarative foundations. Learn why drift happens, how to migrate to a directory-based, trunk-based setup, and how to design for fleet-scale growth.","2026-08-10",[352,353,356,384,385],"argocd","devops",{"path":387,"title":388,"description":389,"date":390,"tags":391},"\u002Fblog\u002Fen\u002Fkubernetes-image-signing-sigstore-supply-chain","Anyone Can Rewrite an Image Tag. Why Kubernetes Needs Sigstore-Backed Signing to Prove Provenance","Container image signing explained: tags can be overwritten by anyone, and passing CI tests doesn't guarantee the image running in production is the one you built. Learn how Sigstore and Kyverno work together to reject unsigned images on Kubernetes\u002FK3s, integrated into a GitOps workflow.","2026-08-06",[352,353,375,356,392],"security",{"path":394,"title":395,"description":396,"date":397,"tags":398},"\u002Fblog\u002Fen\u002Fk3s-edge-fleet-declarative-management","One Device's Troubleshooting Is a Funny Story. A Thousand Devices Is a Business Risk: How Rancher Fleet Rescues K3s Edge Operations from Tribal Knowledge","Fleet management for K3s edge operations breaks down once you're troubleshooting devices one at a time by hand. Here's how declarative management and Rancher Fleet let you design edge operations that don't depend on any single person.","2026-08-03",[352,353,399,356,400],"edge-computing","fleet-management",{"path":402,"title":403,"description":404,"date":405,"tags":406},"\u002Fblog\u002Fen\u002Fplatform-engineering-kubernetes-idp-managed-k3s","Stop Handing Developers Raw Kubernetes: The 'Hiding' Philosophy of Platform Engineering, and Kubo's Answer","An explainer on the relationship between platform engineering and Kubernetes — the design philosophy of shielding developers from K8s complexity, the three pillars of building an IDP, and managed K3s as an alternative.","2026-07-16",[353,352,407,408,356,409],"platform-engineering","internal-developer-platform","managed-kubernetes",1792592635047]