We had set up automated upgrades with K3s's system-upgrade-controller, and before we noticed, the Plan status had been stuck "in progress" for days without moving forward. If you run a multi-node K3s cluster with automation in place, you can run into exactly this kind of trouble: something is stuck, and nobody notices.
This post digs into a problem where system-upgrade-controller, K3s's distinctive automated upgrade mechanism, gets stuck on a specific node and ends up blocking the upgrade of the entire fleet. Once we traced the cause, it came down to three intertwined factors: PodDisruptionBudgets (PDBs), taints, and drain timeouts.
system-upgrade-controller Is Not "Just a CronJob"
First, let's clarify how system-upgrade-controller actually works. It's an OSS tool from Rancher/SUSE where you declaratively define an upgrade policy through a custom resource called Plan, and a controller watches nodes and schedules upgrade Jobs accordingly (K3s official documentation).
If you build standard Kubernetes with kubeadm, this mechanism doesn't exist. kubeadm has no built-in way to automatically roll out cluster upgrades; you either upgrade manually or build your own automation into a CI/CD pipeline. K3s, on the other hand, lets you install system-upgrade-controller and apply a Plan to automatically perform a rolling upgrade, server nodes first and agent nodes second (SUSE official documentation).
How a Plan Behaves
- A node selector specifies which group of nodes to target for the upgrade
- The concurrency field controls how many nodes get upgraded at the same time
- channel (a release channel) or version (a specific version) determines the upgrade target
- It's recommended to split server-node and agent-node Plans, with servers always completing before agents follow
Convenient as it is, if you operate this "Plan → Job → node selection" three-layer structure without understanding it, pinpointing where things are stuck takes time. The official GitHub repository documents in detail how each node's Job gets generated from a Plan (rancher/system-upgrade-controller).
Three Causes Behind a Stuck Plan
When a system-upgrade-controller Plan stops progressing on a given node, it ultimately comes down to one thing: safely draining the target node has stalled somewhere. Let's look at the three most common culprits.
1. A PodDisruptionBudget Blocks the Drain
Kubernetes's PodDisruptionBudget (PDB) protects application availability during voluntary disruptions. When you specify minAvailable or maxUnavailable, the cluster calculates how many disruptions are currently allowed (disruptionsAllowed) (Kubernetes official documentation).
If that value sits at 0, the node's drain process keeps getting 429 Too Many Requests on its eviction requests, and the Pod never gets evicted. This isn't a K3s-specific quirk — it's how the Kubernetes Eviction API works everywhere, since the PDB check happens at the API server's admission-control stage (Kubernetes official documentation: API-initiated Eviction). Microsoft Learn's AKS troubleshooting documentation reports the exact same pattern — getting stuck with ALLOWED DISRUPTIONS at zero (Microsoft Learn). In other words, this is a pitfall common to Kubernetes operations in general, regardless of whether you're self-hosting or using a managed service.
2. Tainted Nodes Are Excluded by Default
The Pods in the upgrade Jobs that system-upgrade-controller generates only tolerate node.kubernetes.io/unschedulable:NoSchedule by default. That means if a node carries any other custom taint — say, one you added for a dedicated workload — the upgrade Job itself won't be scheduled onto that node unless you add an explicit toleration, and the Plan will appear stuck "waiting" (rancher/system-upgrade-controller).
The underlying taint-and-toleration mechanism is standard Kubernetes functionality: NoSchedule only blocks new Pods from being scheduled, while NoExecute also evicts Pods already running there (Kubernetes official documentation: Taints and Tolerations). In the context of system-upgrade-controller, this behavior is easy to overlook, leaving you wondering why one particular node never gets upgraded.
3. The Plan Waits Out a Drain Timeout
Even without PDB or taint issues, if a particular Pod takes a long time to evict for some other reason, the drain process simply waits until it times out. A Plan's drain spec supports options like force, ignoreDaemonSets, and skipWaitForDeleteTimeout, and if you haven't configured these ahead of time, you can end up waiting far longer than expected in production (rancher/system-upgrade-controller).
Of these three factors, we've covered PodDisruptionBudget behavior on its own, from a different angle, in our piece on mass eviction caused by PodDisruptionBudgets in K3s. Here, we're treating it strictly as one of the factors that can stall a system-upgrade-controller Plan.
Recovery and Prevention
Let's separate what to do when a Plan is already stuck from what to set up in advance to prevent it.
Emergency Response (When Already Stuck)
- Run
kubectl get plan -n system-upgradeandkubectl get jobs -n system-upgradeto identify which node's Job is stalled - If a PDB is the cause, run
kubectl get pdb -A, check theALLOWED DISRUPTIONScolumn, and temporarily adjust any PDB sitting at0 - If it still won't progress, delete the Plan once, manually
kubectl uncordonany node left cordoned, and then reapply a corrected Plan
Prevention (Set This Up in Advance)
- Explicitly configure
forceandignoreDaemonSetsin the Plan'sdrainspec instead of relying on the defaults - Inventory the custom taints applied to your upgrade-target nodes, and add tolerations to the Plan's Job spec where needed
- Review whether your PDBs'
minAvailable/maxUnavailablesettings are too strict relative to replica counts before an upgrade - Build Plan status monitoring into your observability stack so you can watch upgrade progress
system-upgrade-controller itself is appealing precisely because installing it and writing a Plan is enough to get it running (K3s official documentation). But behind that convenience lie three pitfalls — PDBs, taints, and drain timeouts — that are worth understanding.
If you'd rather offload the ongoing burden of operating this kind of automation, a managed K3s service can take on the design and monitoring of upgrade operations for you. This matters even more in edge environments with offline or intermittent connectivity, where the lag before you notice a stalled Plan becomes a serious risk. Kubo On-Premise supports air-gapped environments, making it easier to outsource this kind of automation design and monitoring even for edge node fleets on closed networks.
Maybe It's Time to Rethink Your Cluster Setup
If PDB and taint misconfigurations keep tripping up your upgrade operations, that's also a sign it's time to revisit your overall cluster design. With Kubo Cloud's AI-Driven Deployment and Captain UI, you can operate your K3s cluster while visualizing Plan progress and node status, with nothing left as a black box. Because it's compatible with standard Kubernetes, there's no vendor lock-in either, and you can carry your existing AWS/Azure knowledge and assets straight over — which lowers the bar when it's time to rethink how your upgrade automation is designed.
Summary
- K3s's
system-upgrade-controlleris a K3s-specific rolling-upgrade mechanism that doesn't exist in kubeadm-based standard Kubernetes - A Plan getting stuck on a specific node generally comes down to three things: ① a PodDisruptionBudget blocking eviction, ② a taint excluding the node from Job scheduling, and ③ a drain timeout
- For emergencies, delete the Plan and uncordon the node; for the long term, make drain settings explicit and review PDBs/taints ahead of time
- In setups where a stall is hard to notice — like edge environments — it's worth considering a managed service such as Kubo to take upgrade operations off your plate entirely
Automation only earns your trust once you understand how it works. Before you apply your next Plan, run through the three checkpoints covered here.