1. I ran helm upgrade directly in production, and by the next morning the config was gone
An application deployed via K3s's HelmChart CRD needed an urgent parameter change. I ran helm upgrade --set to overwrite the value directly, confirmed the Pod came up with the correct configuration, and moved on.
The next morning, after a routine node maintenance restart, the config had reverted to its state from before the upgrade. helm history showed the revision had incremented, but the Pod actually running was still on the old configuration. Digging into the difference between K3s and Kubernetes, I found the cause wasn't my mistake — it was an automation layer unique to K3s called the "HelmChart CRD."
This kind of anxiety — not knowing what your cluster is automating behind your back — is something worth considering when choosing a K3s-based managed service, too. Services like Kubo emphasize operational transparency, but first you need to understand what K3s itself is doing.
2. The 3-layer structure unique to K3s: HelmChart CRD → helm-controller → Kubernetes Job
In vanilla Kubernetes, running helm install has the Helm client render the chart and create resources directly against the Kubernetes API. There is no "declarative watcher" sitting on top — nothing changes again until someone runs helm upgrade.
K3s bundles something extra: its own Helm Controller. According to K3s's official documentation, K3s uses a Custom Resource Definition (CRD) called HelmChart to declaratively manage the install, upgrade, and uninstall of Helm charts. This isn't limited to bundled components like Traefik or ServiceLB — you can use the same mechanism to auto-deploy your own custom Helm charts as well.
In other words, K3s always has a background structure running: "HelmChart CRD (declaration) → helm-controller (watcher) → Kubernetes Job (execution)." The k3s-io/helm-controller field reference documents many settings such as chartContent and valuesContent, but it does not specify how changes made outside the CRD are handled. That gap is exactly where the trouble starts.
3. What actually happened: failurePolicy and the Job re-trigger
K3s's official documentation defines a failurePolicy setting whose default is "reinstall". When a Job is judged to have failed, this automatically performs a clean uninstall followed by a reinstall. Unless you switch it to "abort", helm-controller re-triggers a Helm-driven Job every time it detects a difference between the desired state (the HelmChart CRD's contents) and the actual state (what's really running in the cluster).
This behavior has been reported as a real-world problem multiple times. helm-controller Issue #50 describes the controller repeatedly re-triggering the install Job and generating new revisions even though the chart version hadn't changed. The logs showed DesiredSet - Replace Wait batch/v1, Kind=Job, indicating the controller kept detecting a need to replace the Job resource.
k3s-io/k3s Issue #2893 documents a similar case: a user had customized Traefik via HelmChartConfig (enabling the dashboard, adding TLS settings), and reported that "every time K3s or the host restarts, this customization isn't picked up and the default Traefik settings apply instead." The workaround mentioned was forcing a re-read by adding whitespace or a newline to the manifest file. From the perspective of this 3-layer structure, a manual kubectl edit is nothing more than "a diff that only survives temporarily."
Whether you can actually trace this Job re-trigger from a management console matters a lot for day-to-day operations. Kubo's Captain UI is designed to surface the state of Helm charts and Jobs running in the background, making it easier to notice when a reconcile has quietly reverted your configuration — reducing the "wait, when did this change back?" moments. Kubo
4. Pitfall #2: some resources don't roll back even after deleting HelmChartConfig
There's another wrinkle: reconcile behavior isn't consistent across every resource type. k3s-io/k3s Issue #5534 reports that after customizing Traefik's ports via HelmChartConfig (to 1234/5555) and then deleting that HelmChartConfig, the LoadBalancer-type Service and its klipper-lb Pod stayed on the custom values instead of reverting to defaults.
In other words, "removing the HelmChartConfig always restores the defaults" isn't a safe assumption either. Both the SUSE community forum and k3s-io/k3s Discussion #11176 show recurring reports of HelmChartConfig either not applying as expected or not reverting properly. K3s's Helm Controller watches /var/lib/rancher/k3s/server/manifests/ at startup and generates Jobs from what it finds there — and the sibling project RKE2's official documentation describes the same mechanism, noting that manifests placed in that directory are auto-deployed as AddOn custom resources. Even so, the diff-detection logic isn't always intuitive on a per-resource-type basis.
5. The right way to fix it: change through HelmChartConfig, and failurePolicy: abort
Given this structure, the rules for changing an app or addon managed by the HelmChart CRD on K3s become clear.
- Don't change things directly with
kubectl editorhelm upgrade --set. Assume it will be wiped out on the next reconcile. - Always make changes through HelmChartConfig's
valuesContent. helm-controller recognizes this as a legitimate input, so it survives reconciliation. - Set
failurePolicy: abortif you want to avoid unintended automatic reinstalls. On failure, this leaves the chart in a failed state awaiting manual intervention instead of silently reinstalling. - When deleting a HelmChartConfig, verify each affected resource actually reverts to its default — as Issue #5534 shows, some resource types don't.
This is exactly the kind of behavior you'd miss if you think of K3s only as "lightweight Kubernetes." helm-controller Issue #351 shows the reconcile logic itself is still an active area of discussion and improvement — this is a structure worth staying alert to going forward.
6. How Kubo changes this
Even on K3s-based Kubo, the HelmChart CRD mechanism itself exists as the same standard K3s feature. The difference is whether this "invisible automation layer" stays a black box or becomes something you can operate with visibility.
Kubo's Captain UI is designed so you can check the state of Helm charts and Jobs running automatically inside the cluster from a management dashboard, making it easier to notice "the reconcile ran and quietly reverted my config" before it becomes a production incident. Automation you'd never see on EKS or AKS becomes something you can operate without a black box on Kubo.
And for finance, healthcare, and manufacturing organizations in air-gapped environments — where even manually editing /var/lib/rancher/k3s/server/manifests/ is difficult — Kubo On-Premise offers an option built around "Your Infrastructure. Your Rules. Your Kubernetes."
7. Summary
The difference between K3s and Kubernetes isn't limited to how ReplicaSets or networking are implemented. The HelmChart CRD + helm-controller + Kubernetes Job structure bundled with K3s is a unique layer that doesn't exist in vanilla Kubernetes — and if you don't know about it, a direct kubectl edit or helm upgrade will quietly get reverted on the next reconcile.
Always make changes through HelmChartConfig, and operate with a clear understanding of how failurePolicy behaves. That's the first step toward running a K3s cluster reliably in production. If you'd rather operate this automation layer with full visibility, Kubo's Captain UI is worth a look.