A Resource Optimizer Broke K3s: All Replicas Evicted at Once. PodDisruptionBudget as the Last Line of Defense

1. One Night, Every Pod in the K3s Cluster Disappeared at Once

To cut costs, we enabled the Vertical Pod Autoscaler (VPA) on a K3s cluster to automatically right-size CPU and memory requests. The results came fast: over-provisioned requests shrank toward actual usage, and free node capacity grew.

Then, one night, the monitoring dashboard filled with 503 errors. Tracing the cause, we found that every replica of a Deployment had been recreated at the same time. Pods that were supposed to roll out one at a time had all vanished at once. Why?

The answer was simple: we had never configured a PodDisruptionBudget (PDB). Left unchecked, a resource optimizer like VPA operates with no safety net. On a small-to-mid-sized cluster like K3s, where replica counts are lower to begin with, the blast radius of this kind of accident is larger. This post explains why "resource optimization" can trigger a total outage, and how PodDisruptionBudget becomes the last line of defense — with concrete configuration examples.

Gaps like this — a missing safety net around an otherwise convenient automation — can happen anywhere a cluster is built and operated by hand. On a managed K3s environment like Kubo, this kind of operational know-how is baked into the standard configuration.

2. Why "Resource Optimization" Can Wipe Out All Replicas — The Pod Eviction Mechanism

VPA is made up of three components: Recommender, Updater, and Admission Controller. According to the official Kubernetes documentation, the updateMode field supports several modes: Off (recommendations are only recorded), Initial (applied only at pod creation), Recreate (pods with a large gap from the recommendation are evicted and recreated), and InPlaceOrRecreate (attempts an in-place update first).

The problem lies in the Recreate-family modes. When a running pod's resource requests drift significantly from the Recommender's suggestion, the Updater evicts that pod and lets it be recreated. The official documentation in the kubernetes/autoscaler repository states plainly that the Updater "uses the Eviction API to evict pods."

The Eviction API is generally treated as a form of "voluntary disruption," as defined by Kubernetes' official documentation on Pod disruptions. If no PodDisruptionBudget exists, there is nothing to stop an eviction triggered through the Eviction API. In other words, if the Updater decides that multiple replicas of the same Deployment need to be evicted, nothing prevents it from acting on all of them at once.

3. Combining It with HPA Makes Things Worse

Many K3s clusters run VPA alongside the Horizontal Pod Autoscaler (HPA) as well, and that combination introduces an even trickier problem. HPA makes scaling decisions based on utilization percentage relative to requests.cpu. That means the moment VPA changes a pod's request value, the calculated utilization percentage shifts even though actual usage hasn't changed at all.

The known limitations documentation in kubernetes/autoscaler is explicit: VPA should not be used together with HPA on the same resource metric. A write-up from ScaleOps describes the resulting mechanism as an infinite loop: VPA lowers requests → HPA's utilization appears higher → excessive scale-out occurs → per-pod usage drops → VPA recommends cutting requests even further.

For a deeper diagnosis of how this HPA/VPA interaction unfolds — including the case where metrics show up as "unknown" — see our earlier post on the HPA/VPA death spiral behind K3s' "unknown metrics" mystery. If you're weighing event-driven alternatives, our guide to choosing between HPA and KEDA is also worth a look. The PDB incident covered here, though, is a more fundamental configuration gap that can occur even without this feedback loop.

4. PodDisruptionBudget: Configuring the "Last Line of Defense"

A PodDisruptionBudget defines an upper limit on how many pods in a replicated application can be down simultaneously due to voluntary disruptions. Per Kubernetes' official Disruptions documentation, you can specify either minAvailable (the minimum number of pods that must stay running) or maxUnavailable (the maximum number of pods allowed to be down at once).

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 75%
  selector:
    matchLabels:
      app: my-app

With this PDB in place, as the official kubernetes/autoscaler documentation confirms, the Updater "respects the pod disruption budget by using the Eviction API," so when an eviction would violate the PDB, it holds off and proceeds one pod at a time instead. As a rule of thumb, keep minAvailable around 50–75%, and never set minAvailable equal to the total replica count — doing so blocks all voluntary disruptions outright.

5. A Production Checklist for Running VPA Safely on K3s

To run VPA safely alongside other autoscalers in a production K3s environment, work through these four checks in order:

• Choose updateMode gradually: Rather than jumping straight to Recreate or InPlaceOrRecreate in production, start with Off or Initial to observe recommendations before enabling automatic updates.
• Always set resourcePolicy.maxAllowed: Cap recommendations so they never exceed a node's maximum capacity. Without this cap, a recommendation that exceeds available node resources can leave a pod stuck Pending indefinitely.
• Always create a PodDisruptionBudget: This is what limits the Updater to a safe, one-at-a-time rolling eviction, as covered in the previous section.
• Separate the metrics HPA and VPA rely on: If HPA scales on CPU/memory, keep VPA in Off/Initial mode. Conversely, if VPA manages CPU/memory automatically, point HPA at custom metrics such as request rate or queue depth instead. Both Google Cloud's GKE documentation on VPA and Microsoft Learn's AKS documentation on VPA recommend this same separation.

It's also worth noting that In-Place Pod Resize — which changes resources without evicting the pod — reached Beta in Kubernetes v1.33 and became Stable (GA) in v1.35. VPA's InPlaceOrRecreate mode can take advantage of this to reduce how often evictions happen at all. That said, on clusters that don't yet support it, evictions still occur the old way — so a PDB remains essential regardless.

Rebuilding this checklist from scratch for every cluster is tedious, unglamorous work. Kubo Cloud ships with Prometheus + Grafana monitoring out of the box, and its Rancher-based management layer lets you track Updater eviction events and VPA recommendation changes across multiple clusters from a single pane of glass. Because it's built on K3s, everything covered here about VPA, HPA, and PDB behavior applies exactly as it would on standard Kubernetes.

6. Summary

The more convenient an automation is, the easier it is to overlook the safety net around it. VPA-driven resource optimization can meaningfully improve production cost efficiency — but deploy it without a PodDisruptionBudget as your last line of defense, and you're carrying the risk that, one night, every replica disappears at once.

If you need to build this kind of PDB design and resource-optimization operation into your own on-premises data center, Kubo On-Premise lets you run something close to a fully managed operation while keeping everything under your own control. If reducing the operational burden of PDB design and eviction control sounds appealing, take a look at our pricing plans for example configurations.