[{"data":1,"prerenderedAt":436},["ShallowReactive",2],{"blog-en-k3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending":3,"blog-related-en-k3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending":377,"blog-en-k3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending-alt":365},{"id":4,"title":5,"author":6,"body":7,"date":359,"description":360,"extension":361,"image":362,"locale":363,"meta":364,"navigation":365,"path":366,"seo":367,"stem":368,"tags":369,"__hash__":376},"blog\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending.md","We Drained a Node and a Pod Never Came Back. Why Topology Spread Constraints and PodDisruptionBudget Collide on K3s","Kubo Team",{"type":8,"value":9,"toc":348},"minimark",[10,15,23,26,46,50,56,80,87,91,97,114,138,167,171,177,197,203,207,213,226,248,252,258,267,283,286,309,318,322,325,335],[11,12,14],"h2",{"id":13},"_1-the-incident-a-routine-node-drain-left-a-pod-behind","1. The Incident — A Routine Node Drain Left a Pod Behind",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Fsection01.webp",[16,24,25],{},"During a routine monthly OS patch cycle on a K3s cluster, we cordoned and drained nodes one at a time, as always. Pods on the target node evicted cleanly to the other nodes, and most workloads recovered within seconds. But one Pod from a single Deployment stayed stuck in a Pending state and never came back.",[16,27,28,32,33,37,38,45],{},[29,30,31],"code",{},"kubectl get pods"," showed the PodDisruptionBudget was configured correctly, and eviction itself should have been permitted. Yet the new Pod wasn't being scheduled anywhere. Someone on the team suspected a misconfigured PDB, but the real cause turned out to live somewhere else entirely. This article verifies, based on official documentation, how a Pod can stay stuck even when ",[34,35,36],"strong",{},"both Topology Spread Constraints and PodDisruptionBudget are working exactly as designed",". Chasing down this kind of configuration interaction by hand every time it happens is a real burden for infrastructure engineers — it's exactly the kind of thing ",[39,40,44],"a",{"href":41,"rel":42},"https:\u002F\u002Fkubo.hexabase.io\u002F",[43],"nofollow","Kubo","'s Captain UI, which visualizes cluster state, is meant to take off your plate.",[11,47,49],{"id":48},"_2-investigation-the-pdb-was-fine-the-scheduler-was-the-problem","2. Investigation — The PDB Was Fine. The Scheduler Was the Problem",[16,51,52],{},[19,53],{"alt":54,"src":55},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Fsection02.webp",[16,57,58,59,62,63,68,69,72,73,76,77],{},"The first suspect was the PodDisruptionBudget. According to the Kubernetes documentation, the Eviction API evaluates the PDB and returns ",[29,60,61],{},"429 Too Many Requests"," when an eviction isn't currently permitted (",[39,64,67],{"href":65,"rel":66},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Fapi-eviction\u002F",[43],"Kubernetes: API-initiated Eviction","). But ",[29,70,71],{},"kubectl get pdb"," showed ",[29,74,75],{},"disruptionsAllowed"," at 1 or more — the eviction itself had gone through fine. The Pod had already been evicted correctly; ",[34,78,79],{},"a new Pod simply wasn't being scheduled.",[16,81,82,83,86],{},"Checking ",[29,84,85],{},"kubectl get events --sort-by='.lastTimestamp'"," revealed scheduling-failure events pointing to a topology skew violation. The culprit wasn't the PDB — it was the Topology Spread Constraints configured on the same Deployment.",[11,88,90],{"id":89},"_3-the-mechanism-when-maxskew-and-donotschedule-become-a-hard-constraint","3. The Mechanism — When maxSkew and DoNotSchedule Become a Hard Constraint",[16,92,93],{},[19,94],{"alt":95,"src":96},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Fsection03.webp",[16,98,99,100,105,106,109,110,113],{},"According to ",[39,101,104],{"href":102,"rel":103},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Ftopology-spread-constraints",[43],"the Kubernetes documentation",", ",[29,107,108],{},"maxSkew"," defines \"the maximum allowed difference between the number of matching Pods in the target topology and the minimum matching Pod count elsewhere.\" ",[29,111,112],{},"whenUnsatisfiable"," has two values:",[115,116,117,130],"ul",{},[118,119,120,125,126,129],"li",{},[34,121,122],{},[29,123,124],{},"DoNotSchedule"," (hard constraint): if the condition can't be met, the Pod is ",[34,127,128],{},"not scheduled"," and stays Pending",[118,131,132,137],{},[34,133,134],{},[29,135,136],{},"ScheduleAnyway"," (soft constraint): the scheduler prefers placements that reduce skew but always schedules the Pod",[16,139,140,141,105,144,105,147,150,151,156,157,159,160,162,163,166],{},"Our configuration was ",[29,142,143],{},"topologyKey: kubernetes.io\u002Fhostname",[29,145,146],{},"maxSkew: 1",[29,148,149],{},"whenUnsatisfiable: DoNotSchedule",". The ",[39,152,155],{"href":153,"rel":154},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fprescriptive-guidance\u002Flatest\u002Fha-resiliency-amazon-eks-apps\u002Fspread-workloads.html",[43],"AWS Prescriptive Guidance"," also describes ",[29,158,124],{}," combined with ",[29,161,108],{}," and ",[29,164,165],{},"minDomains"," as the pattern for building a hard requirement. The question was how that hard requirement could break down mid-drain.",[11,168,170],{"id":169},"_4-the-core-issue-draining-created-its-own-node-shortage","4. The Core Issue — Draining Created Its Own Node Shortage",[16,172,173],{},[19,174],{"alt":175,"src":176},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Fsection04.webp",[16,178,179,180,183,184,186,187,190,191,196],{},"The technical knowledge base StackHarbor calls this pattern the \"Pending-Forever Trap.\" With a ",[29,181,182],{},"maxSkew:1"," hostname constraint under ",[29,185,124],{},", if ",[34,188,189],{},"replica count exceeds node count",", the placement request becomes structurally unsatisfiable for the scheduler, and the Pod stays Pending indefinitely (",[39,192,195],{"href":193,"rel":194},"https:\u002F\u002Fstackharbor.com\u002Fen\u002Fknowledge-base\u002Fk8s-topology-spread-constraints\u002F",[43],"StackHarbor: Kubernetes Topology Spread Constraints",").",[16,198,199,200,202],{},"That's exactly what happened here. Under normal conditions, 3 nodes held 3 replicas, one per node — a perfectly even spread. The moment maintenance cordoned one node, the effective pool of schedulable nodes dropped to 2. With 3 replicas against 2 nodes, no placement could satisfy ",[29,201,182],{}," anymore, and the evicted Pod had nowhere to go. PodDisruptionBudget only governs \"is it okay to delete this,\" not \"where does it go afterward.\" The drain operation itself had undermined the precondition Topology Spread Constraints depended on.",[11,204,206],{"id":205},"_5-why-poddisruptionbudget-alone-doesnt-protect-you-and-other-pitfalls-to-watch","5. Why PodDisruptionBudget Alone Doesn't Protect You — And Other Pitfalls to Watch",[16,208,209],{},[19,210],{"alt":211,"src":212},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Fsection05.webp",[16,214,215,216,219,220,225],{},"What this incident reveals is that PodDisruptionBudget and Topology Spread Constraints are ",[34,217,218],{},"independent features solving different problems",". CAST AI's engineering blog notes that the scheduler doesn't inherently track every zone in a cluster by default, and when a node pool scales to zero in an autoscaling environment, the topology domain itself can disappear, leaving no node that satisfies the constraint (",[39,221,224],{"href":222,"rel":223},"https:\u002F\u002Fcast.ai\u002Fblog\u002Ftopology-spread-constraints-for-increased-cluster-availability-and-efficiency-and-a-much-better-cost\u002F",[43],"CAST AI: Topology Spread Constraints","). A node drain creates the same situation temporarily.",[16,227,228,229,231,232,235,236,241,242,247],{},"Relaxing to ",[29,230,136],{}," (a soft constraint) isn't a free pass either, because it's only evaluated at scheduling time — if the distribution drifts afterward, already-running Pods are never automatically rebalanced. The official Kubernetes Descheduler project addresses this with its ",[29,233,234],{},"RemovePodsViolatingTopologySpreadConstraint"," strategy, which evicts only the Pods violating the topology rule after the fact. But that strategy still follows the principle of never evicting a Pod in a way that would violate its PDB, so all three mechanisms need to be combined with a clear understanding of which layer each one protects (",[39,237,240],{"href":238,"rel":239},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fdescheduler",[43],"kubernetes-sigs\u002Fdescheduler","). Interestingly, the original KEP (Kubernetes Enhancement Proposal) notes that Pod Anti-Affinity's strict \"at most one Pod per domain\" rule was itself prone to blocking placement during rolling updates — a problem Topology Spread Constraints was designed to solve (",[39,243,246],{"href":244,"rel":245},"https:\u002F\u002Fwww.kubernetes.dev\u002Fresources\u002Fkeps\u002F895\u002F",[43],"KEP-895: Pod Topology Spread","). It's an ironic twist that the feature built to avoid Anti-Affinity's problem reproduces a similar one when combined with node draining.",[11,249,251],{"id":250},"_6-putting-this-into-practice-for-k3s-node-maintenance","6. Putting This Into Practice for K3s Node Maintenance",[16,253,254],{},[19,255],{"alt":256,"src":257},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Fsection06.webp",[16,259,260,261,266],{},"The official K3s documentation recommends manually draining and cordoning a node before an upgrade if your workload is sensitive to brief API server outages, and explicitly requires upgrading server nodes one at a time first, then agent nodes (",[39,262,265],{"href":263,"rel":264},"https:\u002F\u002Fdocs.k3s.io\u002Fupgrades\u002Fmanual",[43],"K3s: Manual Upgrades","). That \"one at a time\" unit of work is exactly what creates the momentary node shortage.",[16,268,269,270,273,274,277,278,196],{},"If you're running Longhorn for storage, there's an additional wrinkle. Its documentation recommends setting ",[29,271,272],{},"kubectl drain --timeout"," high enough (or unlimited) for replica rebuilds to finish, and requires that the ",[34,275,276],{},"number of nodes upgraded concurrently stay lower than each volume's replica count"," (",[39,279,282],{"href":280,"rel":281},"https:\u002F\u002Flonghorn.io\u002Fdocs\u002F1.9.2\u002Fmaintenance\u002Fmaintenance\u002F",[43],"Longhorn: Node Maintenance",[16,284,285],{},"Three practical mitigations are worth adopting:",[115,287,288,300,306],{},[118,289,290,291,293,294,296,297,299],{},"Temporarily relax the affected Deployment's ",[29,292,112],{}," from ",[29,295,124],{}," to ",[29,298,136],{}," right before maintenance, then restore it afterward",[118,301,302,303,305],{},"Revisit ",[29,304,108],{}," against the actual ratio of replica count to node count",[118,307,308],{},"Build a maintenance checklist that accounts for both Topology Spread Constraints and PodDisruptionBudget together",[16,310,311,312,317],{},"Standardizing procedures like this tends to rely on tribal knowledge. If you need to run maintenance reproducibly on-premises or in air-gapped environments, a fully self-managed K3s platform like ",[39,313,316],{"href":314,"rel":315},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[43],"Kubo On-Premise"," is worth considering.",[11,319,321],{"id":320},"_7-summary","7. Summary",[16,323,324],{},"PodDisruptionBudget and Topology Spread Constraints both protect availability, but they protect different things. PDB governs \"how many Pods are allowed to go down,\" while Topology Spread Constraints governs \"where a Pod is allowed to land.\" An operation like node draining, which temporarily changes the cluster's own denominator, shakes both preconditions at once.",[16,326,327,328,330,331,334],{},"The key insight here is that before suspecting a PDB misconfiguration, it pays to connect the obvious fact that \"node count temporarily drops during a drain\" with the ",[29,329,108],{}," calculation. With that lens, you can get from a scheduling-failure event in ",[29,332,333],{},"kubectl get events"," to the root cause much faster.",[16,336,337,338,341,342,347],{},"If you'd rather not spend engineering time chasing down configuration interactions like this and standardizing maintenance procedures on a K3s-based production cluster, ",[39,339,44],{"href":41,"rel":340},[43],"'s AI-Driven Deployment and Captain UI visualization can help. If you're interested, reach out via our ",[39,343,346],{"href":344,"rel":345},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[43],"contact form",".",{"title":349,"searchDepth":350,"depth":350,"links":351},"",2,[352,353,354,355,356,357,358],{"id":13,"depth":350,"text":14},{"id":48,"depth":350,"text":49},{"id":89,"depth":350,"text":90},{"id":169,"depth":350,"text":170},{"id":205,"depth":350,"text":206},{"id":250,"depth":350,"text":251},{"id":320,"depth":350,"text":321},"2026-10-19","Draining a node in production K3s left a Pod stuck Pending. The cause: Topology Spread Constraints' maxSkew colliding with PodDisruptionBudget. We verify how draining itself creates a node shortage, based on official documentation.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending",{"title":5,"description":360},"blog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending",[370,371,372,373,374,375],"k3s","kubernetes","topology-spread-constraints","poddisruptionbudget","scheduling","node-drain","UuBnFB175yDpLm-YK3I6UV_6anE00nE9TCgn0TE5qks",[378,387,396,406,418,428],{"path":379,"title":380,"description":381,"date":382,"tags":383},"\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction","A Resource Optimizer Broke K3s: All Replicas Evicted at Once. PodDisruptionBudget as the Last Line of Defense","A real incident where enabling automated resource optimization on K3s caused all replicas to be evicted simultaneously, triggering a full outage. This post explains why pods get evicted en masse and how PodDisruptionBudget's minAvailable setting prevents it.","2026-09-18",[370,371,373,384,385,386],"eviction","vertical-scaling","resource-optimization",{"path":388,"title":389,"description":390,"date":391,"tags":392},"\u002Fblog\u002Fen\u002Fbroadcast-redundancy-kubernetes-high-availability-design","Why Does the FIFA World Cup Broadcast Never Go Dark? What Broadcast Engineering's Dual-Path Redundancy Teaches Us About Real Kubernetes High Availability","Kubernetes high availability isn't achieved simply by adding more replicas. Using the broadcast industry's SMPTE ST 2022-7 dual-path transmission as a lens, this article unpacks what Topology Spread Constraints and multi-AZ design actually mean.","2026-07-20",[371,370,393,372,394,395],"high-availability","sre","managed-kubernetes",{"path":397,"title":398,"description":399,"date":400,"tags":401},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[370,371,402,403,404,405],"hpa","containerresource","autoscaling","service-mesh",{"path":407,"title":408,"description":409,"date":410,"tags":411},"\u002Fblog\u002Fen\u002Fk3s-vs-kubernetes-cni-flannel-default-comparison","K3s vs Standard Kubernetes: How Different Their CNI Starting Points Really Are","K3s ships with Flannel built in, while standard Kubernetes starts with no CNI selected at all. This comparison, grounded in the official K3s, Kubernetes, Calico, and Cilium docs, explains how that difference affects NetworkPolicy enforcement and network performance.","2026-10-20",[370,371,412,413,414,415,416,417],"cni","flannel","calico","cilium","networkpolicy","networking",{"path":419,"title":420,"description":421,"date":422,"tags":423},"\u002Fblog\u002Fen\u002Fkyverno-validatingadmissionpolicy-cel-migration-silent-bypass","Is the CEL Rewrite of Your Policy Really the Same? How Migrating from Kyverno to ValidatingAdmissionPolicy Let Some Resources Silently Slip Through","As Kyverno's ClusterPolicy heads toward deprecation, teams migrating to ValidatingAdmissionPolicy (CEL) have found that validation silently skipped for certain resource types. Here's the CEL runtime error and failurePolicy trap behind it, and a safe migration path.","2026-10-18",[370,371,424,425,426,427],"validatingadmissionpolicy","kyverno","cel","admission-control",{"path":429,"title":430,"description":431,"date":432,"tags":433},"\u002Fblog\u002Fen\u002Fk3s-vpa-inplaceorrecreate-pod-resize-cgroup","VPA InPlaceOrRecreate Didn't Stop the Restart. Testing InPlacePodVerticalScaling on K3s","Does VPA's InPlaceOrRecreate mode and In-Place Pod Resize really avoid Pod restarts? We test QoS class constraints and the cgroup v1 trap on K3s with real configs and kubectl commands.","2026-10-17",[370,371,434,404,435],"vpa","cgroup",1792592640140]