[{"data":1,"prerenderedAt":519},["ShallowReactive",2],{"blog-en-k3s-vpa-poddisruptionbudget-mass-eviction":3,"blog-related-en-k3s-vpa-poddisruptionbudget-mass-eviction":459,"blog-en-k3s-vpa-poddisruptionbudget-mass-eviction-alt":447},{"id":4,"title":5,"author":6,"body":7,"date":441,"description":442,"extension":443,"image":444,"locale":445,"meta":446,"navigation":447,"path":448,"seo":449,"stem":450,"tags":451,"__hash__":458},"blog\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction.md","A Resource Optimizer Broke K3s: All Replicas Evicted at Once. PodDisruptionBudget as the Last Line of Defense","Kubo Team",{"type":8,"value":9,"toc":433},"minimark",[10,15,19,27,34,45,49,56,86,98,111,115,121,131,146,159,163,169,185,288,306,310,316,319,382,399,407,411,414,429],[11,12,14],"h2",{"id":13},"_1-one-night-every-pod-in-the-k3s-cluster-disappeared-at-once","1. One Night, Every Pod in the K3s Cluster Disappeared at Once",[16,17,18],"p",{},"To cut costs, we enabled the Vertical Pod Autoscaler (VPA) on a K3s cluster to automatically right-size CPU and memory requests. The results came fast: over-provisioned requests shrank toward actual usage, and free node capacity grew.",[16,20,21,22,26],{},"Then, one night, the monitoring dashboard filled with 503 errors. Tracing the cause, we found that ",[23,24,25],"strong",{},"every replica of a Deployment had been recreated at the same time."," Pods that were supposed to roll out one at a time had all vanished at once. Why?",[16,28,29,30,33],{},"The answer was simple: ",[23,31,32],{},"we had never configured a PodDisruptionBudget (PDB)."," Left unchecked, a resource optimizer like VPA operates with no safety net. On a small-to-mid-sized cluster like K3s, where replica counts are lower to begin with, the blast radius of this kind of accident is larger. This post explains why \"resource optimization\" can trigger a total outage, and how PodDisruptionBudget becomes the last line of defense — with concrete configuration examples.",[16,35,36,37,44],{},"Gaps like this — a missing safety net around an otherwise convenient automation — can happen anywhere a cluster is built and operated by hand. On a managed K3s environment like ",[38,39,43],"a",{"href":40,"rel":41},"https:\u002F\u002Fkubo.hexabase.io\u002F",[42],"nofollow","Kubo",", this kind of operational know-how is baked into the standard configuration.",[11,46,48],{"id":47},"_2-why-resource-optimization-can-wipe-out-all-replicas-the-pod-eviction-mechanism","2. Why \"Resource Optimization\" Can Wipe Out All Replicas — The Pod Eviction Mechanism",[16,50,51],{},[52,53],"img",{"alt":54,"src":55},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction\u002Fsection01.webp",[16,57,58,59,64,65,69,70,73,74,77,78,81,82,85],{},"VPA is made up of three components: Recommender, Updater, and Admission Controller. According to the ",[38,60,63],{"href":61,"rel":62},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fautoscaling\u002Fvertical-pod-autoscale\u002F",[42],"official Kubernetes documentation",", the ",[66,67,68],"code",{},"updateMode"," field supports several modes: ",[66,71,72],{},"Off"," (recommendations are only recorded), ",[66,75,76],{},"Initial"," (applied only at pod creation), ",[66,79,80],{},"Recreate"," (pods with a large gap from the recommendation are evicted and recreated), and ",[66,83,84],{},"InPlaceOrRecreate"," (attempts an in-place update first).",[16,87,88,89,91,92,97],{},"The problem lies in the ",[66,90,80],{},"-family modes. When a running pod's resource requests drift significantly from the Recommender's suggestion, the Updater evicts that pod and lets it be recreated. The ",[38,93,96],{"href":94,"rel":95},"https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fautoscaler\u002Fblob\u002Fmaster\u002Fvertical-pod-autoscaler\u002Fdocs\u002Fcomponents.md",[42],"official documentation in the kubernetes\u002Fautoscaler repository"," states plainly that the Updater \"uses the Eviction API to evict pods.\"",[16,99,100,101,106,107,110],{},"The Eviction API is generally treated as a form of \"voluntary disruption,\" as defined by ",[38,102,105],{"href":103,"rel":104},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fdisruptions\u002F",[42],"Kubernetes' official documentation on Pod disruptions",". ",[23,108,109],{},"If no PodDisruptionBudget exists, there is nothing to stop an eviction triggered through the Eviction API."," In other words, if the Updater decides that multiple replicas of the same Deployment need to be evicted, nothing prevents it from acting on all of them at once.",[11,112,114],{"id":113},"_3-combining-it-with-hpa-makes-things-worse","3. Combining It with HPA Makes Things Worse",[16,116,117],{},[52,118],{"alt":119,"src":120},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction\u002Fsection02.webp",[16,122,123,124,127,128],{},"Many K3s clusters run VPA alongside the Horizontal Pod Autoscaler (HPA) as well, and that combination introduces an even trickier problem. HPA makes scaling decisions based on utilization percentage relative to ",[66,125,126],{},"requests.cpu",". That means ",[23,129,130],{},"the moment VPA changes a pod's request value, the calculated utilization percentage shifts even though actual usage hasn't changed at all.",[16,132,133,134,139,140,145],{},"The ",[38,135,138],{"href":136,"rel":137},"https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fautoscaler\u002Fblob\u002Fmaster\u002Fvertical-pod-autoscaler\u002Fdocs\u002Fknown-limitations.md",[42],"known limitations documentation in kubernetes\u002Fautoscaler"," is explicit: VPA should not be used together with HPA on the same resource metric. A ",[38,141,144],{"href":142,"rel":143},"https:\u002F\u002Fscaleops.com\u002Fblog\u002Fhpas-three-architectural-flaws-and-why-your-autoscaling-keeps-failing\u002F",[42],"write-up from ScaleOps"," describes the resulting mechanism as an infinite loop: VPA lowers requests → HPA's utilization appears higher → excessive scale-out occurs → per-pod usage drops → VPA recommends cutting requests even further.",[16,147,148,149,153,154,158],{},"For a deeper diagnosis of how this HPA\u002FVPA interaction unfolds — including the case where metrics show up as \"unknown\" — see our earlier post on ",[38,150,152],{"href":151},"\u002Fblog\u002Fja\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral","the HPA\u002FVPA death spiral behind K3s' \"unknown metrics\" mystery",". If you're weighing event-driven alternatives, ",[38,155,157],{"href":156},"\u002Fblog\u002Fja\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","our guide to choosing between HPA and KEDA"," is also worth a look. The PDB incident covered here, though, is a more fundamental configuration gap that can occur even without this feedback loop.",[11,160,162],{"id":161},"_4-poddisruptionbudget-configuring-the-last-line-of-defense","4. PodDisruptionBudget: Configuring the \"Last Line of Defense\"",[16,164,165],{},[52,166],{"alt":167,"src":168},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction\u002Fsection03.webp",[16,170,171,172,176,177,180,181,184],{},"A PodDisruptionBudget defines an upper limit on how many pods in a replicated application can be down simultaneously due to voluntary disruptions. Per ",[38,173,175],{"href":103,"rel":174},[42],"Kubernetes' official Disruptions documentation",", you can specify either ",[66,178,179],{},"minAvailable"," (the minimum number of pods that must stay running) or ",[66,182,183],{},"maxUnavailable"," (the maximum number of pods allowed to be down at once).",[186,187,192],"pre",{"className":188,"code":189,"language":190,"meta":191,"style":191},"language-yaml shiki shiki-themes tokyo-night","apiVersion: policy\u002Fv1\nkind: PodDisruptionBudget\nmetadata:\n  name: my-app-pdb\nspec:\n  minAvailable: 75%\n  selector:\n    matchLabels:\n      app: my-app\n","yaml","",[66,193,194,211,222,231,242,250,261,269,277],{"__ignoreMap":191},[195,196,199,203,207],"span",{"class":197,"line":198},"line",1,[195,200,202],{"class":201},"s0U2E","apiVersion",[195,204,206],{"class":205},"sAklC",":",[195,208,210],{"class":209},"sPY7s"," policy\u002Fv1\n",[195,212,214,217,219],{"class":197,"line":213},2,[195,215,216],{"class":201},"kind",[195,218,206],{"class":205},[195,220,221],{"class":209}," PodDisruptionBudget\n",[195,223,225,228],{"class":197,"line":224},3,[195,226,227],{"class":201},"metadata",[195,229,230],{"class":205},":\n",[195,232,234,237,239],{"class":197,"line":233},4,[195,235,236],{"class":201},"  name",[195,238,206],{"class":205},[195,240,241],{"class":209}," my-app-pdb\n",[195,243,245,248],{"class":197,"line":244},5,[195,246,247],{"class":201},"spec",[195,249,230],{"class":205},[195,251,253,256,258],{"class":197,"line":252},6,[195,254,255],{"class":201},"  minAvailable",[195,257,206],{"class":205},[195,259,260],{"class":209}," 75%\n",[195,262,264,267],{"class":197,"line":263},7,[195,265,266],{"class":201},"  selector",[195,268,230],{"class":205},[195,270,272,275],{"class":197,"line":271},8,[195,273,274],{"class":201},"    matchLabels",[195,276,230],{"class":205},[195,278,280,283,285],{"class":197,"line":279},9,[195,281,282],{"class":201},"      app",[195,284,206],{"class":205},[195,286,287],{"class":209}," my-app\n",[16,289,290,291,295,296,298,299,305],{},"With this PDB in place, as the ",[38,292,294],{"href":94,"rel":293},[42],"official kubernetes\u002Fautoscaler documentation"," confirms, the Updater \"respects the pod disruption budget by using the Eviction API,\" so when an eviction would violate the PDB, it holds off and proceeds one pod at a time instead. As a rule of thumb, keep ",[66,297,179],{}," around 50–75%, and ",[23,300,301,302,304],{},"never set ",[66,303,179],{}," equal to the total replica count"," — doing so blocks all voluntary disruptions outright.",[11,307,309],{"id":308},"_5-a-production-checklist-for-running-vpa-safely-on-k3s","5. A Production Checklist for Running VPA Safely on K3s",[16,311,312],{},[52,313],{"alt":314,"src":315},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction\u002Fsection04.webp",[16,317,318],{},"To run VPA safely alongside other autoscalers in a production K3s environment, work through these four checks in order:",[320,321,322,343,352,358],"ul",{},[323,324,325,331,332,334,335,337,338,334,340,342],"li",{},[23,326,327,328,330],{},"Choose ",[66,329,68],{}," gradually",": Rather than jumping straight to ",[66,333,80],{}," or ",[66,336,84],{}," in production, start with ",[66,339,72],{},[66,341,76],{}," to observe recommendations before enabling automatic updates.",[323,344,345,351],{},[23,346,347,348],{},"Always set ",[66,349,350],{},"resourcePolicy.maxAllowed",": Cap recommendations so they never exceed a node's maximum capacity. Without this cap, a recommendation that exceeds available node resources can leave a pod stuck Pending indefinitely.",[323,353,354,357],{},[23,355,356],{},"Always create a PodDisruptionBudget",": This is what limits the Updater to a safe, one-at-a-time rolling eviction, as covered in the previous section.",[323,359,360,363,364,366,367,369,370,375,376,381],{},[23,361,362],{},"Separate the metrics HPA and VPA rely on",": If HPA scales on CPU\u002Fmemory, keep VPA in ",[66,365,72],{},"\u002F",[66,368,76],{}," mode. Conversely, if VPA manages CPU\u002Fmemory automatically, point HPA at custom metrics such as request rate or queue depth instead. Both ",[38,371,374],{"href":372,"rel":373},"https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fverticalpodautoscaler",[42],"Google Cloud's GKE documentation on VPA"," and ",[38,377,380],{"href":378,"rel":379},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Faks\u002Fvertical-pod-autoscaler",[42],"Microsoft Learn's AKS documentation on VPA"," recommend this same separation.",[16,383,384,385,375,390,395,396,398],{},"It's also worth noting that In-Place Pod Resize — which changes resources without evicting the pod — ",[38,386,389],{"href":387,"rel":388},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2025\u002F05\u002F16\u002Fkubernetes-v1-33-in-place-pod-resize-beta\u002F",[42],"reached Beta in Kubernetes v1.33",[38,391,394],{"href":392,"rel":393},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2025\u002F12\u002F19\u002Fkubernetes-v1-35-in-place-pod-resize-ga",[42],"became Stable (GA) in v1.35",". VPA's ",[66,397,84],{}," mode can take advantage of this to reduce how often evictions happen at all. That said, on clusters that don't yet support it, evictions still occur the old way — so a PDB remains essential regardless.",[16,400,401,402,406],{},"Rebuilding this checklist from scratch for every cluster is tedious, unglamorous work. ",[38,403,405],{"href":40,"rel":404},[42],"Kubo Cloud"," ships with Prometheus + Grafana monitoring out of the box, and its Rancher-based management layer lets you track Updater eviction events and VPA recommendation changes across multiple clusters from a single pane of glass. Because it's built on K3s, everything covered here about VPA, HPA, and PDB behavior applies exactly as it would on standard Kubernetes.",[11,408,410],{"id":409},"_6-summary","6. Summary",[16,412,413],{},"The more convenient an automation is, the easier it is to overlook the safety net around it. VPA-driven resource optimization can meaningfully improve production cost efficiency — but deploy it without a PodDisruptionBudget as your last line of defense, and you're carrying the risk that, one night, every replica disappears at once.",[16,415,416,417,422,423,428],{},"If you need to build this kind of PDB design and resource-optimization operation into your own on-premises data center, ",[38,418,421],{"href":419,"rel":420},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[42],"Kubo On-Premise"," lets you run something close to a fully managed operation while keeping everything under your own control. If reducing the operational burden of PDB design and eviction control sounds appealing, take a look at our ",[38,424,427],{"href":425,"rel":426},"https:\u002F\u002Fwww.hexabase.com\u002Fpricing\u002F",[42],"pricing plans"," for example configurations.",[430,431,432],"style",{},"html pre.shiki code .s0U2E, html code.shiki .s0U2E{--shiki-default:#F7768E}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":191,"searchDepth":213,"depth":213,"links":434},[435,436,437,438,439,440],{"id":13,"depth":213,"text":14},{"id":47,"depth":213,"text":48},{"id":113,"depth":213,"text":114},{"id":161,"depth":213,"text":162},{"id":308,"depth":213,"text":309},{"id":409,"depth":213,"text":410},"2026-09-18","A real incident where enabling automated resource optimization on K3s caused all replicas to be evicted simultaneously, triggering a full outage. This post explains why pods get evicted en masse and how PodDisruptionBudget's minAvailable setting prevents it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction",{"title":5,"description":442},"blog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction",[452,453,454,455,456,457],"k3s","kubernetes","poddisruptionbudget","eviction","vertical-scaling","resource-optimization","but57WWODORfO6goq5912_riH5kxWagIBfKTiGVE74U",[460,469,479,491,501,509],{"path":461,"title":462,"description":463,"date":464,"tags":465},"\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending","We Drained a Node and a Pod Never Came Back. Why Topology Spread Constraints and PodDisruptionBudget Collide on K3s","Draining a node in production K3s left a Pod stuck Pending. The cause: Topology Spread Constraints' maxSkew colliding with PodDisruptionBudget. We verify how draining itself creates a node shortage, based on official documentation.","2026-10-19",[452,453,466,454,467,468],"topology-spread-constraints","scheduling","node-drain",{"path":470,"title":471,"description":472,"date":473,"tags":474},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[452,453,475,476,477,478],"hpa","containerresource","autoscaling","service-mesh",{"path":480,"title":481,"description":482,"date":483,"tags":484},"\u002Fblog\u002Fen\u002Fk3s-vs-kubernetes-cni-flannel-default-comparison","K3s vs Standard Kubernetes: How Different Their CNI Starting Points Really Are","K3s ships with Flannel built in, while standard Kubernetes starts with no CNI selected at all. This comparison, grounded in the official K3s, Kubernetes, Calico, and Cilium docs, explains how that difference affects NetworkPolicy enforcement and network performance.","2026-10-20",[452,453,485,486,487,488,489,490],"cni","flannel","calico","cilium","networkpolicy","networking",{"path":492,"title":493,"description":494,"date":495,"tags":496},"\u002Fblog\u002Fen\u002Fkyverno-validatingadmissionpolicy-cel-migration-silent-bypass","Is the CEL Rewrite of Your Policy Really the Same? How Migrating from Kyverno to ValidatingAdmissionPolicy Let Some Resources Silently Slip Through","As Kyverno's ClusterPolicy heads toward deprecation, teams migrating to ValidatingAdmissionPolicy (CEL) have found that validation silently skipped for certain resource types. Here's the CEL runtime error and failurePolicy trap behind it, and a safe migration path.","2026-10-18",[452,453,497,498,499,500],"validatingadmissionpolicy","kyverno","cel","admission-control",{"path":502,"title":503,"description":504,"date":505,"tags":506},"\u002Fblog\u002Fen\u002Fk3s-vpa-inplaceorrecreate-pod-resize-cgroup","VPA InPlaceOrRecreate Didn't Stop the Restart. Testing InPlacePodVerticalScaling on K3s","Does VPA's InPlaceOrRecreate mode and In-Place Pod Resize really avoid Pod restarts? We test QoS class constraints and the cgroup v1 trap on K3s with real configs and kubectl commands.","2026-10-17",[452,453,507,477,508],"vpa","cgroup",{"path":510,"title":511,"description":512,"date":513,"tags":514},"\u002Fblog\u002Fen\u002Fk3s-pod-security-admission-restricted-servicelb-outage","We Enabled Restricted and ServiceLB Died: The K3s Pod Security Admission Gap No Generic Guide Mentions","Enabling the restricted Pod Security Admission profile on K3s following standard Kubernetes guides can silently kill ServiceLB. Here's why, backed by K3s's own CIS hardening guide, and how to roll it out safely.","2026-10-16",[452,453,515,516,517,518],"pod-security-admission","pod-security-standards","servicelb","security",1792592640967]