A K3s "Patch-Only" Update Broke Our containerd Config Template. The v1-to-v2 Migration Happens Outside the Version Skew Policy

1. How It Started — A K3s Upgrade That Was Supposed to Be "Just a Patch"

There's an instinct that most Kubernetes operators develop over time: "Be careful with minor version upgrades. But a patch release? That's just bug fixes and security patches — go ahead and apply it right away."

Consider a team running a K3s cluster with customized containerd behavior — registry mirror settings and a custom runtime handler, written into containerd's config template (config.toml.tmpl). One day, they rolled out a patch-level K3s upgrade through their automated pipeline. Only a single digit in the version number changed. Yet after the upgrade, they found their custom containerd configuration was no longer being applied.

Tracing the root cause, they discovered that while the Kubernetes minor version hadn't changed at all, the bundled containerd had jumped a full major version underneath it. This isn't some environment-specific fluke — it's a documented, known change recorded in K3s's own official release notes.

2. Kubernetes' Official Version Skew Policy Only Talks About Minor Versions

Let's start with the baseline. Kubernetes has an official version skew policy that spells out exactly how much version difference is tolerated between kubelet and kube-apiserver. The key points:

• kubelet must never be newer than kube-apiserver
• As of Kubernetes 1.25, kubelet can be up to 3 minor versions older than kube-apiserver
• For older Kubernetes releases, only 2 minor versions of skew are supported

This "n-3" tolerance originated as KEP-3935, which was implemented as a stable feature in v1.28. The KEP's goal is straightforward: node upgrades are more disruptive to workloads than control-plane upgrades, so the proposal widens the window so nodes can lag further behind before they must catch up.

What matters here is that this policy only covers version differences in the Kubernetes core itself. There are clear rules for kubelet and kube-apiserver versions, but the policy says nothing at all about the versions of the surrounding components a distribution bundles — the container runtime, the CNI plugin, DNS, the ingress controller, and so on.

3. A Real Case: containerd Jumped From v1 to v2 in a Patch Release

Let's back up the opening scenario with an actual example. According to the official K3s GitHub release, v1.32.1+k3s1 (released January 28, 2025) bundled containerd v1.7.23-k3s2.

But the very next release, v1.32.2+k3s1 (released February 27, 2025), switched to containerd v2.0.2-k3s2. The Kubernetes version string only moved from v1.32.1 to v1.32.2 — a single patch-number increment. And yet the official release notes themselves state that containerd was "bumped to version 2.0" and that a "new config file schema" was introduced, requiring users with custom configurations to migrate their templates.

In other words, this wasn't a bug — it was a change K3s had officially announced. But that announcement never appears anywhere near "the Kubernetes version skew policy," which is the resource operators typically consult. If you don't read the fine print of the release notes, it's entirely understandable to treat a patch-number bump as a "minor update" at face value.

4. containerd 2.0's Breaking Changes — What Quietly Breaks

The official containerd documentation describes the major breaking changes introduced in 2.0:

• Docker Schema 1 image pulls are disabled by default (the application/vnd.docker.distribution.manifest.v1+prettyjws format)
• The legacy runtime shims deprecated back in containerd 1.4 (io.containerd.runtime.v1.linux and io.containerd.runc.v1) are completely removed, making migration to io.containerd.runc.v2 mandatory
• io_uring-related syscalls are disallowed by default
• Legacy components such as the CRI v1alpha2 API and the AUFS snapshotter are removed

Beyond a missed config template migration, environments still relying on old-format container images, or environments that explicitly reference a legacy runtime handler, can see this surface as image pull failures or container start failures.

By the same logic, the current K3s v1.36 line keeps its bundled component set changing with every release. For example, v1.36.5+k3s1 bundles containerd v2.3.4-k3s1.36, etcd v3.6.14-k3s3, Flannel v0.28.4, CoreDNS v1.14.7, Traefik v3.7.13, and local-path-provisioner v0.0.37. It's worth remembering just how many components are bundled behind that single version string.

5. A Safe K3s Upgrade Procedure — What the Official Docs Already Warned About

To be fair, K3s's own upgrade guide builds risk mitigation into its recommended procedure from the start:

• Don't skip minor versions in your upgrade plan (to stay within the Kubernetes version skew policy)
• Upgrade server nodes one at a time, then agent nodes
• If you have workloads sensitive to a brief API server interruption, manually drain and cordon the node beforehand, then uncordon it after the upgrade

That procedure is correct as far as it goes. The problem is that the "skew policy" this procedure protects you against, and the kind of bundled-component incompatibility we saw with containerd, are simply different layers of the problem. The former is about minor-version differences in Kubernetes core itself; the latter is about how a distribution manages the versions of the dependencies it chose to bundle. Following one faithfully doesn't protect you from an incident caused by the other.

Nodes becoming unresponsive or unstable during a cluster upgrade isn't unique to K3s, either. Red Hat's knowledge base, for instance, documents cases where OpenShift nodes get stuck in a NotReady state during cluster upgrades due to kubelet certificate or configuration mismatches. The causes vary, but the unglamorous discipline of "read the release notes end-to-end on every upgrade" remains just as important regardless.

If you're running K3s, it's worth making it a hard rule to check the "Urgent Upgrade Notes" and changelog sections of the release notes on every upgrade, patch releases included. And if you've automated upgrades through a CI pipeline, it's worth asking whether that check itself can be automated too.

6. Conclusion — Who Carries the Burden of "a Verified Combination"?

Following Kubernetes' official version skew policy and having a safe K3s patch release are two different things. K3s distributes not just the Kubernetes core, but bundled peripheral components — containerd, Flannel, CoreDNS, Traefik, local-path-provisioner — all under a single version string. Behind that convenience lies room for incompatible changes that a patch number alone can't tell you about.

Whether your own operations team absorbs all of that verification work, or you get it delivered as a pre-verified combination, is a decision that directly affects your operating cost. Kubo is a managed Kubernetes service built on K3s, and Hexabase verifies the combination of K3s version and bundled components before it ships — so operators don't have to individually read through every release note on every upgrade.

If the cost of verifying "is this patch actually safe?" on every K3s upgrade is adding up for your team, it's worth comparing Kubo Cloud's verified upgrade path. Beyond the cost structure compared to EKS or AKS, factoring in this "verification effort" changes how the total cost of ownership looks. Combined with AI-driven deployment and the visualized Captain UI, you can see the concrete comparison on the pricing page.

And if data sovereignty or security requirements mean you can't hand off your own environment, Kubo On-Premise lets you apply the same verified version combinations on-premises. If you'd like to talk through an upgrade strategy, feel free to reach out via contact us.