[{"data":1,"prerenderedAt":507},["ShallowReactive",2],{"blog-en-k3s-system-upgrade-controller-plan-stuck-node-drain":3,"blog-related-en-k3s-system-upgrade-controller-plan-stuck-node-drain":455,"blog-en-k3s-system-upgrade-controller-plan-stuck-node-drain-alt":445},{"id":4,"title":5,"author":6,"body":7,"date":439,"description":440,"extension":441,"image":442,"locale":443,"meta":444,"navigation":445,"path":446,"seo":447,"stem":448,"tags":449,"__hash__":454},"blog\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain.md","The Plan Never Disappeared. Why K3s's system-upgrade-controller Got Stuck on One Node","Kubo Team",{"type":8,"value":9,"toc":420},"minimark",[10,23,29,34,41,62,76,81,113,121,125,131,141,145,169,200,204,217,237,241,263,275,279,285,288,292,325,329,357,369,378,382,391,395,417],[11,12,13,14,18,19,22],"p",{},"We had set up automated upgrades with K3s's ",[15,16,17],"code",{},"system-upgrade-controller",", and before we noticed, the ",[15,20,21],{},"Plan"," status had been stuck \"in progress\" for days without moving forward. If you run a multi-node K3s cluster with automation in place, you can run into exactly this kind of trouble: something is stuck, and nobody notices.",[11,24,25,26,28],{},"This post digs into a problem where ",[15,27,17],{},", K3s's distinctive automated upgrade mechanism, gets stuck on a specific node and ends up blocking the upgrade of the entire fleet. Once we traced the cause, it came down to three intertwined factors: PodDisruptionBudgets (PDBs), taints, and drain timeouts.",[30,31,33],"h2",{"id":32},"system-upgrade-controller-is-not-just-a-cronjob","system-upgrade-controller Is Not \"Just a CronJob\"",[11,35,36],{},[37,38],"img",{"alt":39,"src":40},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain\u002Fsection01.webp",[11,42,43,44,46,47,49,50,53,54,61],{},"First, let's clarify how ",[15,45,17],{}," actually works. It's an OSS tool from Rancher\u002FSUSE where you declaratively define an upgrade policy through a custom resource called ",[15,48,21],{},", and a controller watches nodes and schedules upgrade ",[15,51,52],{},"Job","s accordingly (",[55,56,60],"a",{"href":57,"rel":58},"https:\u002F\u002Fdocs.k3s.io\u002Fupgrades\u002Fautomated",[59],"nofollow","K3s official documentation",").",[11,63,64,65,67,68,70,71,61],{},"If you build standard Kubernetes with kubeadm, this mechanism doesn't exist. kubeadm has no built-in way to automatically roll out cluster upgrades; you either upgrade manually or build your own automation into a CI\u002FCD pipeline. K3s, on the other hand, lets you install ",[15,66,17],{}," and apply a ",[15,69,21],{}," to automatically perform a rolling upgrade, server nodes first and agent nodes second (",[55,72,75],{"href":73,"rel":74},"https:\u002F\u002Fdocumentation.suse.com\u002Fcloudnative\u002Fk3s\u002Flatest\u002Fen\u002Fupgrades\u002Fautomated.html",[59],"SUSE official documentation",[77,78,80],"h3",{"id":79},"how-a-plan-behaves","How a Plan Behaves",[82,83,84,93,100,110],"ul",{},[85,86,87,88,92],"li",{},"A ",[89,90,91],"strong",{},"node selector"," specifies which group of nodes to target for the upgrade",[85,94,95,96,99],{},"The ",[89,97,98],{},"concurrency"," field controls how many nodes get upgraded at the same time",[85,101,102,105,106,109],{},[89,103,104],{},"channel"," (a release channel) or ",[89,107,108],{},"version"," (a specific version) determines the upgrade target",[85,111,112],{},"It's recommended to split server-node and agent-node Plans, with servers always completing before agents follow",[11,114,115,116,61],{},"Convenient as it is, if you operate this \"Plan → Job → node selection\" three-layer structure without understanding it, pinpointing where things are stuck takes time. The official GitHub repository documents in detail how each node's Job gets generated from a Plan (",[55,117,120],{"href":118,"rel":119},"https:\u002F\u002Fgithub.com\u002Francher\u002Fsystem-upgrade-controller",[59],"rancher\u002Fsystem-upgrade-controller",[30,122,124],{"id":123},"three-causes-behind-a-stuck-plan","Three Causes Behind a Stuck Plan",[11,126,127],{},[37,128],{"alt":129,"src":130},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain\u002Fsection02.webp",[11,132,133,134,136,137,140],{},"When a ",[15,135,17],{}," Plan stops progressing on a given node, it ultimately comes down to one thing: ",[89,138,139],{},"safely draining the target node has stalled somewhere",". Let's look at the three most common culprits.",[77,142,144],{"id":143},"_1-a-poddisruptionbudget-blocks-the-drain","1. A PodDisruptionBudget Blocks the Drain",[11,146,147,148,151,152,155,156,159,160,163,164,61],{},"Kubernetes's ",[15,149,150],{},"PodDisruptionBudget"," (PDB) protects application availability during voluntary disruptions. When you specify ",[15,153,154],{},"minAvailable"," or ",[15,157,158],{},"maxUnavailable",", the cluster calculates how many disruptions are currently allowed (",[15,161,162],{},"disruptionsAllowed",") (",[55,165,168],{"href":166,"rel":167},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Frun-application\u002Fconfigure-pdb\u002F",[59],"Kubernetes official documentation",[11,170,171,172,175,176,179,180,185,186,189,190,195,196,199],{},"If that value sits at ",[15,173,174],{},"0",", the node's drain process keeps getting ",[15,177,178],{},"429 Too Many Requests"," on its eviction requests, and the Pod never gets evicted. This isn't a K3s-specific quirk — it's how the Kubernetes Eviction API works everywhere, since the PDB check happens at the API server's admission-control stage (",[55,181,184],{"href":182,"rel":183},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Fapi-eviction\u002F",[59],"Kubernetes official documentation: API-initiated Eviction","). Microsoft Learn's AKS troubleshooting documentation reports the exact same pattern — getting stuck with ",[15,187,188],{},"ALLOWED DISRUPTIONS"," at zero (",[55,191,194],{"href":192,"rel":193},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Ftroubleshoot\u002Fazure\u002Fazure-kubernetes\u002Fcreate-upgrade-delete\u002Ferror-code-poddrainfailure",[59],"Microsoft Learn","). In other words, this is ",[89,197,198],{},"a pitfall common to Kubernetes operations in general",", regardless of whether you're self-hosting or using a managed service.",[77,201,203],{"id":202},"_2-tainted-nodes-are-excluded-by-default","2. Tainted Nodes Are Excluded by Default",[11,205,206,207,209,210,213,214,61],{},"The Pods in the upgrade Jobs that ",[15,208,17],{}," generates only tolerate ",[15,211,212],{},"node.kubernetes.io\u002Funschedulable:NoSchedule"," by default. That means if a node carries any other custom taint — say, one you added for a dedicated workload — the upgrade Job itself won't be scheduled onto that node unless you add an explicit toleration, and the Plan will appear stuck \"waiting\" (",[55,215,120],{"href":118,"rel":216},[59],[11,218,219,220,223,224,227,228,233,234,236],{},"The underlying taint-and-toleration mechanism is standard Kubernetes functionality: ",[15,221,222],{},"NoSchedule"," only blocks new Pods from being scheduled, while ",[15,225,226],{},"NoExecute"," also evicts Pods already running there (",[55,229,232],{"href":230,"rel":231},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Ftaint-and-toleration\u002F",[59],"Kubernetes official documentation: Taints and Tolerations","). In the context of ",[15,235,17],{},", this behavior is easy to overlook, leaving you wondering why one particular node never gets upgraded.",[77,238,240],{"id":239},"_3-the-plan-waits-out-a-drain-timeout","3. The Plan Waits Out a Drain Timeout",[11,242,243,244,247,248,251,252,255,256,259,260,61],{},"Even without PDB or taint issues, if a particular Pod takes a long time to evict for some other reason, the drain process simply waits until it times out. A Plan's ",[15,245,246],{},"drain"," spec supports options like ",[15,249,250],{},"force",", ",[15,253,254],{},"ignoreDaemonSets",", and ",[15,257,258],{},"skipWaitForDeleteTimeout",", and if you haven't configured these ahead of time, you can end up waiting far longer than expected in production (",[55,261,120],{"href":118,"rel":262},[59],[11,264,265,266,271,272,274],{},"Of these three factors, we've covered PodDisruptionBudget behavior on its own, from a different angle, in ",[55,267,270],{"href":268,"rel":269},"https:\u002F\u002Fkubo.hexabase.io\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction",[59],"our piece on mass eviction caused by PodDisruptionBudgets in K3s",". Here, we're treating it strictly as one of the factors that can stall a ",[15,273,17],{}," Plan.",[30,276,278],{"id":277},"recovery-and-prevention","Recovery and Prevention",[11,280,281],{},[37,282],{"alt":283,"src":284},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain\u002Fsection03.webp",[11,286,287],{},"Let's separate what to do when a Plan is already stuck from what to set up in advance to prevent it.",[77,289,291],{"id":290},"emergency-response-when-already-stuck","Emergency Response (When Already Stuck)",[293,294,295,306,318],"ol",{},[85,296,297,298,301,302,305],{},"Run ",[15,299,300],{},"kubectl get plan -n system-upgrade"," and ",[15,303,304],{},"kubectl get jobs -n system-upgrade"," to identify which node's Job is stalled",[85,307,308,309,312,313,315,316],{},"If a PDB is the cause, run ",[15,310,311],{},"kubectl get pdb -A",", check the ",[15,314,188],{}," column, and temporarily adjust any PDB sitting at ",[15,317,174],{},[85,319,320,321,324],{},"If it still won't progress, delete the Plan once, manually ",[15,322,323],{},"kubectl uncordon"," any node left cordoned, and then reapply a corrected Plan",[77,326,328],{"id":327},"prevention-set-this-up-in-advance","Prevention (Set This Up in Advance)",[82,330,331,342,345,354],{},[85,332,333,334,301,336,338,339,341],{},"Explicitly configure ",[15,335,250],{},[15,337,254],{}," in the Plan's ",[15,340,246],{}," spec instead of relying on the defaults",[85,343,344],{},"Inventory the custom taints applied to your upgrade-target nodes, and add tolerations to the Plan's Job spec where needed",[85,346,347,348,350,351,353],{},"Review whether your PDBs' ",[15,349,154],{},"\u002F",[15,352,158],{}," settings are too strict relative to replica counts before an upgrade",[85,355,356],{},"Build Plan status monitoring into your observability stack so you can watch upgrade progress",[11,358,359,361,362,364,365,368],{},[15,360,17],{}," itself is appealing precisely because installing it and writing a ",[15,363,21],{}," is enough to get it running (",[55,366,60],{"href":57,"rel":367},[59],"). But behind that convenience lie three pitfalls — PDBs, taints, and drain timeouts — that are worth understanding.",[11,370,371,372,377],{},"If you'd rather offload the ongoing burden of operating this kind of automation, a managed K3s service can take on the design and monitoring of upgrade operations for you. This matters even more in edge environments with offline or intermittent connectivity, where the lag before you notice a stalled Plan becomes a serious risk. ",[55,373,376],{"href":374,"rel":375},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[59],"Kubo On-Premise"," supports air-gapped environments, making it easier to outsource this kind of automation design and monitoring even for edge node fleets on closed networks.",[30,379,381],{"id":380},"maybe-its-time-to-rethink-your-cluster-setup","Maybe It's Time to Rethink Your Cluster Setup",[11,383,384,385,390],{},"If PDB and taint misconfigurations keep tripping up your upgrade operations, that's also a sign it's time to revisit your overall cluster design. With ",[55,386,389],{"href":387,"rel":388},"https:\u002F\u002Fkubo.hexabase.io\u002F",[59],"Kubo Cloud","'s AI-Driven Deployment and Captain UI, you can operate your K3s cluster while visualizing Plan progress and node status, with nothing left as a black box. Because it's compatible with standard Kubernetes, there's no vendor lock-in either, and you can carry your existing AWS\u002FAzure knowledge and assets straight over — which lowers the bar when it's time to rethink how your upgrade automation is designed.",[30,392,394],{"id":393},"summary","Summary",[82,396,397,403,406,409],{},[85,398,399,400,402],{},"K3s's ",[15,401,17],{}," is a K3s-specific rolling-upgrade mechanism that doesn't exist in kubeadm-based standard Kubernetes",[85,404,405],{},"A Plan getting stuck on a specific node generally comes down to three things: ① a PodDisruptionBudget blocking eviction, ② a taint excluding the node from Job scheduling, and ③ a drain timeout",[85,407,408],{},"For emergencies, delete the Plan and uncordon the node; for the long term, make drain settings explicit and review PDBs\u002Ftaints ahead of time",[85,410,411,412,416],{},"In setups where a stall is hard to notice — like edge environments — it's worth considering a managed service such as ",[55,413,415],{"href":387,"rel":414},[59],"Kubo"," to take upgrade operations off your plate entirely",[11,418,419],{},"Automation only earns your trust once you understand how it works. Before you apply your next Plan, run through the three checkpoints covered here.",{"title":421,"searchDepth":422,"depth":422,"links":423},"",2,[424,428,433,437,438],{"id":32,"depth":422,"text":33,"children":425},[426],{"id":79,"depth":427,"text":80},3,{"id":123,"depth":422,"text":124,"children":429},[430,431,432],{"id":143,"depth":427,"text":144},{"id":202,"depth":427,"text":203},{"id":239,"depth":427,"text":240},{"id":277,"depth":422,"text":278,"children":434},[435,436],{"id":290,"depth":427,"text":291},{"id":327,"depth":427,"text":328},{"id":380,"depth":422,"text":381},{"id":393,"depth":422,"text":394},"2026-10-31","We set up automated upgrades with K3s's system-upgrade-controller, and the Plan stopped progressing on a single node. The culprits were PodDisruptionBudgets and taints. Here's how the mechanism works, how to recover, and how Kubo helps.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain",{"title":5,"description":440},"blog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain",[450,451,17,452,453],"k3s","kubernetes","rolling-upgrade","edge-computing","yQxjkaoPlBucAPBF6cpNp9FKgOkOKVN42rIURv70mcQ",[456,464,473,481,490,498],{"path":457,"title":458,"description":459,"date":460,"tags":461},"\u002Fblog\u002Fen\u002Fk3s-prometheus-agent-mode-stale-marker-edge-monitoring","We Switched K3s Edge Prometheus to Agent Mode. The Moment a Target Disappeared, So Did Its Metrics","Running Prometheus Agent mode on K3s edge clusters can leave stale values in your central dashboard when a target disappears, depending on how stale markers are handled. Here's how to design K3s edge monitoring correctly.","2026-11-03",[450,451,462,453,463],"prometheus","observability",{"path":465,"title":466,"description":467,"date":468,"tags":469},"\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join","K3s's Embedded Registry Mirror Was Supposed to Just Work. Then a New Edge Node Couldn't Pull Images","K3s's embedded registry mirror (Spegel) is supposed to work cluster-wide, but a newly added edge node alone failed to pull images. The cause: an undistributed registries.yaml, credential drift, and a known containerd 2.0 bug. Here's the correct distribution workflow for air-gapped operations.","2026-10-30",[450,451,470,471,472,453],"spegel","registry-mirror","airgap",{"path":474,"title":475,"description":476,"date":477,"tags":478},"\u002Fblog\u002Fen\u002Fk3s-microshift-etcd-ostree-edge-comparison","Is Pinning etcd at the Edge a Trap or a Trade-off? K3s vs MicroShift and the Real Cost of OSTree Auto-Rollback","K3s and MicroShift are both called \"lightweight Kubernetes,\" but their assumptions about datastores and OS update models are fundamentally different. We compare etcd lock-in and OSTree auto-rollback to help you choose for production edge deployments.","2026-10-29",[450,479,451,453,480],"microshift","redhat",{"path":482,"title":483,"description":484,"date":485,"tags":486},"\u002Fblog\u002Fen\u002Fk3s-vs-microk8s-ha-etcd-dqlite-edge-production","K3s vs MicroK8s: Which One Actually Wins in Production? The HA Fork Between dqlite and etcd","K3s vs MicroK8s: both are called 'lightweight Kubernetes,' but the real production divide is HA architecture. We compare K3s's embedded etcd and MicroK8s's dqlite using official documentation data.","2026-10-25",[450,487,451,488,453,489],"microk8s","high-availability","comparison",{"path":491,"title":492,"description":493,"date":494,"tags":495},"\u002Fblog\u002Fen\u002Fedge-k3s-hardware-selection-beyond-appliance","Homelabbers Are Ditching ISP Routers for Self-Built PCs. Edge Kubernetes Should Rethink Its Appliance Habit Too","A growing number of individuals are replacing ISP-supplied routers with self-built mini PCs running open-source firmware. The same logic applies to edge Kubernetes: a hardware selection checklist and a GitOps operating model for standardizing dedicated appliances out of the fleet.","2026-08-29",[450,451,453,496,497],"hardware-selection","network-appliance",{"path":499,"title":500,"description":501,"date":502,"tags":503},"\u002Fblog\u002Fen\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium","Bare-Metal K3s LoadBalancer: When ARP Hits Its Limit, Design for BGP","When you can't rely on a cloud load balancer for on-prem or edge K3s clusters, why does Layer2 (ARP) load balancing eventually hit a wall? A practical look at MetalLB's BGP mode vs. Cilium's BGP Control Plane, and how to decide when to migrate.","2026-08-26",[450,451,504,505,506,453],"metallb","bgp","load-balancer",1793798727408]