1. The Incident — A Routine Node Drain Left a Pod Behind
During a routine monthly OS patch cycle on a K3s cluster, we cordoned and drained nodes one at a time, as always. Pods on the target node evicted cleanly to the other nodes, and most workloads recovered within seconds. But one Pod from a single Deployment stayed stuck in a Pending state and never came back.
kubectl get pods showed the PodDisruptionBudget was configured correctly, and eviction itself should have been permitted. Yet the new Pod wasn't being scheduled anywhere. Someone on the team suspected a misconfigured PDB, but the real cause turned out to live somewhere else entirely. This article verifies, based on official documentation, how a Pod can stay stuck even when both Topology Spread Constraints and PodDisruptionBudget are working exactly as designed. Chasing down this kind of configuration interaction by hand every time it happens is a real burden for infrastructure engineers — it's exactly the kind of thing Kubo's Captain UI, which visualizes cluster state, is meant to take off your plate.
2. Investigation — The PDB Was Fine. The Scheduler Was the Problem
The first suspect was the PodDisruptionBudget. According to the Kubernetes documentation, the Eviction API evaluates the PDB and returns 429 Too Many Requests when an eviction isn't currently permitted (Kubernetes: API-initiated Eviction). But kubectl get pdb showed disruptionsAllowed at 1 or more — the eviction itself had gone through fine. The Pod had already been evicted correctly; a new Pod simply wasn't being scheduled.
Checking kubectl get events --sort-by='.lastTimestamp' revealed scheduling-failure events pointing to a topology skew violation. The culprit wasn't the PDB — it was the Topology Spread Constraints configured on the same Deployment.
3. The Mechanism — When maxSkew and DoNotSchedule Become a Hard Constraint
According to the Kubernetes documentation, maxSkew defines "the maximum allowed difference between the number of matching Pods in the target topology and the minimum matching Pod count elsewhere." whenUnsatisfiable has two values:
DoNotSchedule(hard constraint): if the condition can't be met, the Pod is not scheduled and stays PendingScheduleAnyway(soft constraint): the scheduler prefers placements that reduce skew but always schedules the Pod
Our configuration was topologyKey: kubernetes.io/hostname, maxSkew: 1, whenUnsatisfiable: DoNotSchedule. The AWS Prescriptive Guidance also describes DoNotSchedule combined with maxSkew and minDomains as the pattern for building a hard requirement. The question was how that hard requirement could break down mid-drain.
4. The Core Issue — Draining Created Its Own Node Shortage
The technical knowledge base StackHarbor calls this pattern the "Pending-Forever Trap." With a maxSkew:1 hostname constraint under DoNotSchedule, if replica count exceeds node count, the placement request becomes structurally unsatisfiable for the scheduler, and the Pod stays Pending indefinitely (StackHarbor: Kubernetes Topology Spread Constraints).
That's exactly what happened here. Under normal conditions, 3 nodes held 3 replicas, one per node — a perfectly even spread. The moment maintenance cordoned one node, the effective pool of schedulable nodes dropped to 2. With 3 replicas against 2 nodes, no placement could satisfy maxSkew:1 anymore, and the evicted Pod had nowhere to go. PodDisruptionBudget only governs "is it okay to delete this," not "where does it go afterward." The drain operation itself had undermined the precondition Topology Spread Constraints depended on.
5. Why PodDisruptionBudget Alone Doesn't Protect You — And Other Pitfalls to Watch
What this incident reveals is that PodDisruptionBudget and Topology Spread Constraints are independent features solving different problems. CAST AI's engineering blog notes that the scheduler doesn't inherently track every zone in a cluster by default, and when a node pool scales to zero in an autoscaling environment, the topology domain itself can disappear, leaving no node that satisfies the constraint (CAST AI: Topology Spread Constraints). A node drain creates the same situation temporarily.
Relaxing to ScheduleAnyway (a soft constraint) isn't a free pass either, because it's only evaluated at scheduling time — if the distribution drifts afterward, already-running Pods are never automatically rebalanced. The official Kubernetes Descheduler project addresses this with its RemovePodsViolatingTopologySpreadConstraint strategy, which evicts only the Pods violating the topology rule after the fact. But that strategy still follows the principle of never evicting a Pod in a way that would violate its PDB, so all three mechanisms need to be combined with a clear understanding of which layer each one protects (kubernetes-sigs/descheduler). Interestingly, the original KEP (Kubernetes Enhancement Proposal) notes that Pod Anti-Affinity's strict "at most one Pod per domain" rule was itself prone to blocking placement during rolling updates — a problem Topology Spread Constraints was designed to solve (KEP-895: Pod Topology Spread). It's an ironic twist that the feature built to avoid Anti-Affinity's problem reproduces a similar one when combined with node draining.
6. Putting This Into Practice for K3s Node Maintenance
The official K3s documentation recommends manually draining and cordoning a node before an upgrade if your workload is sensitive to brief API server outages, and explicitly requires upgrading server nodes one at a time first, then agent nodes (K3s: Manual Upgrades). That "one at a time" unit of work is exactly what creates the momentary node shortage.
If you're running Longhorn for storage, there's an additional wrinkle. Its documentation recommends setting kubectl drain --timeout high enough (or unlimited) for replica rebuilds to finish, and requires that the number of nodes upgraded concurrently stay lower than each volume's replica count (Longhorn: Node Maintenance).
Three practical mitigations are worth adopting:
- Temporarily relax the affected Deployment's
whenUnsatisfiablefromDoNotScheduletoScheduleAnywayright before maintenance, then restore it afterward - Revisit
maxSkewagainst the actual ratio of replica count to node count - Build a maintenance checklist that accounts for both Topology Spread Constraints and PodDisruptionBudget together
Standardizing procedures like this tends to rely on tribal knowledge. If you need to run maintenance reproducibly on-premises or in air-gapped environments, a fully self-managed K3s platform like Kubo On-Premise is worth considering.
7. Summary
PodDisruptionBudget and Topology Spread Constraints both protect availability, but they protect different things. PDB governs "how many Pods are allowed to go down," while Topology Spread Constraints governs "where a Pod is allowed to land." An operation like node draining, which temporarily changes the cluster's own denominator, shakes both preconditions at once.
The key insight here is that before suspecting a PDB misconfiguration, it pays to connect the obvious fact that "node count temporarily drops during a drain" with the maxSkew calculation. With that lens, you can get from a scheduling-failure event in kubectl get events to the root cause much faster.
If you'd rather not spend engineering time chasing down configuration interactions like this and standardizing maintenance procedures on a K3s-based production cluster, Kubo's AI-Driven Deployment and Captain UI visualization can help. If you're interested, reach out via our contact form.