[{"data":1,"prerenderedAt":293},["ShallowReactive",2],{"blog-en-k3s-gpu-oomkilled-vpa-resource-estimation":3,"blog-related-en-k3s-gpu-oomkilled-vpa-resource-estimation":246,"blog-en-k3s-gpu-oomkilled-vpa-resource-estimation-alt":233},{"id":4,"title":5,"author":6,"body":7,"date":227,"description":228,"extension":229,"image":230,"locale":231,"meta":232,"navigation":233,"path":234,"seo":235,"stem":236,"tags":237,"__hash__":245},"blog\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation.md","GPU Memory Ran Out and Killed an AI Training Job. Why VPA Can't Save You on K3s","Kubo Team",{"type":8,"value":9,"toc":218},"minimark",[10,15,19,27,31,34,41,52,88,91,100,104,107,113,134,141,145,158,164,173,194,203,207,210],[11,12,14],"h2",{"id":13},"a-job-submitted-at-night-vanished-without-a-trace-by-morning","A Job Submitted at Night Vanished Without a Trace by Morning",[16,17,18],"p",{},"You submit a GPU fine-tuning job to a shared Kubernetes cluster and check on it the next morning, only to find it was OOMKilled (forcibly terminated due to insufficient memory). Plenty of operators have hit this exact pattern. The log shows nothing but a single \"Killed\" line, and hours of compute cost are gone.",[16,20,21,22,26],{},"What makes this tricky is that the usual fix for CPU or memory pods — \"just bump up the requests a bit\" — ",[23,24,25],"strong",{},"doesn't work for GPUs",". GPUs are treated specially within Kubernetes, and if you design your resource allocation without understanding that special treatment, you'll repeat the same incident.",[11,28,30],{"id":29},"why-padding-your-requests-doesnt-solve-it","Why \"Padding Your Requests\" Doesn't Solve It",[16,32,33],{},"CPU and memory requests\u002Flimits can be set relatively flexibly. CPU is throttled gently, and memory can absorb some variation if you set requests with a bit of headroom.",[16,35,36],{},[37,38],"img",{"alt":39,"src":40},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation\u002Fsection01.webp",[16,42,43,44,51],{},"GPUs are a different story. The ",[45,46,50],"a",{"href":47,"rel":48},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fmanage-gpus\u002Fscheduling-gpus\u002F",[49],"nofollow","official Kubernetes documentation"," spells out the rules for specifying GPU resources:",[53,54,55,68,80],"ul",{},[56,57,58,59,63,64,67],"li",{},"You can set GPU ",[60,61,62],"code",{},"limits"," without ",[60,65,66],{},"requests"," — Kubernetes will use the limit value as the request by default",[56,69,70,71,73,74,76,77],{},"If you specify both ",[60,72,62],{}," and ",[60,75,66],{},", ",[23,78,79],{},"the two values must be equal",[56,81,82,83,85,86],{},"You cannot specify GPU ",[60,84,66],{}," without also specifying ",[60,87,62],{},[16,89,90],{},"In other words, GPUs have none of the \"keep requests low, limits high\" buffering that CPU allows. Whatever GPU a Pod requests is exclusively occupied, and any slack isn't shared with other Pods. That's exactly why a single miscalculation in your requests estimate translates directly into an OOMKilled event or a scheduling failure.",[16,92,93,94,99],{},"On top of that, the ",[45,95,98],{"href":96,"rel":97},"https:\u002F\u002Fwww.fairwinds.com\u002Fblog\u002Fhow-do-i-run-ai-workloads-on-kubernetes-without-wasting-gpus",[49],"Fairwinds blog"," notes that \"fleet average GPU utilization is often well below 30 percent, with recent data putting it as low as 5 percent.\" In many production environments, GPU resource sizing gets decided once at deploy time and is never revisited.",[11,101,103],{"id":102},"the-inconvenient-truth-you-cant-hand-this-off-to-vpa","The Inconvenient Truth: You Can't Hand This Off to VPA",[16,105,106],{},"\"Just let autoscaling handle it\" doesn't hold up for GPU workloads.",[16,108,109],{},[37,110],{"alt":111,"src":112},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation\u002Fsection02.webp",[16,114,115,116,121,122,127,128,133],{},"Check the ",[45,117,120],{"href":118,"rel":119},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fautoscaling\u002Fvertical-pod-autoscale\u002F",[49],"official Kubernetes Vertical Pod Autoscaler documentation"," and you'll find the Recommender consistently analyzes \"CPU and memory\" — GPU is never mentioned. ",[45,123,126],{"href":124,"rel":125},"https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fverticalpodautoscaler",[49],"Google Cloud's GKE documentation"," states the same scope: \"by default, vertical Pod autoscaling computes recommendations for both CPU and memory.\" ",[45,129,132],{"href":130,"rel":131},"https:\u002F\u002Fscaleops.com\u002Fblog\u002Fkubernetes-vpa\u002F",[49],"ScaleOps's VPA explainer"," likewise frames the resource parameters VPA manages — requests and limits — consistently around CPU and memory.",[16,135,136,137,140],{},"Teams that have automated their operations with HPA and VPA are especially prone to assuming \"GPU will get auto-optimized the same way.\" But GPU resource sizing remains, for now, ",[23,138,139],{},"a domain that still requires humans to measure and design manually",".",[11,142,144],{"id":143},"so-what-should-your-estimates-be-based-on-a-measurement-driven-design-process","So What Should Your Estimates Be Based On? A Measurement-Driven Design Process",[16,146,147,148,153,154,157],{},"If you can't rely on automation, you have to rely on measured data. ",[45,149,152],{"href":150,"rel":151},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-02-20-kubernetes-resource-requests-limits\u002Fview",[49],"OneUptime's resource sizing guide"," suggests setting CPU requests to the median (P50) and memory requests to the P99 of actual measured usage, with memory limits set at 1.5-2x requests. Since GPU forces requests to equal limits, the \"give limits some headroom\" approach doesn't directly apply — but the underlying sequence, ",[23,155,156],{},"measure first, then decide the numbers",", still carries over.",[16,159,160],{},[37,161],{"alt":162,"src":163},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation\u002Fsection03.webp",[16,165,166,167,172],{},"For the measurement layer, a setup that continuously visualizes GPU utilization at the node and Pod level with Prometheus + Grafana is the foundation. ",[45,168,171],{"href":169,"rel":170},"https:\u002F\u002Fkubo.hexabase.io\u002F",[49],"Kubo Cloud"," bundles this Prometheus + Grafana monitoring standard into its K3s-based setup, so you can start measuring GPU usage without building a separate monitoring stack.",[16,174,175,176,181,182,187,188,193],{},"As operations mature, the challenge shifts from sizing alone to allocation — who gets how much GPU. ",[45,177,180],{"href":178,"rel":179},"https:\u002F\u002Fkueue.sigs.k8s.io\u002F",[49],"Kueue",", a CNCF-hosted project, provides quota management and job queueing for batch and AI\u002FML workloads, and can dynamically borrow capacity from idle resource pools. A ",[45,183,186],{"href":184,"rel":185},"https:\u002F\u002Fwww.cncf.io\u002Fblog\u002F2026\u002F08\u002F27\u002Fbuilding-an-ai-factory-on-kubernetes\u002F",[49],"CNCF blog post"," from August 2026 describes an \"AI factory\" operating model where multiple teams safely share the same GPU pool, with a combination of technologies like DRA and Kueue being key to the implementation. The mechanism that exposes GPUs to Nodes and Pods in the first place is handled by ",[45,189,192],{"href":190,"rel":191},"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fk8s-device-plugin",[49],"NVIDIA's official device plugin"," — understanding these layers before designing your requests\u002Flimits is the shortest path to preventing OOMKilled incidents.",[16,195,196,197,202],{},"If you're planning for large-scale multi-tenant operations, it's worth considering a platform like ",[45,198,201],{"href":199,"rel":200},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[49],"Kubo On-Premise",", where you can measure and tune GPU resources incrementally on your own infrastructure. This matters especially for industries that can't send data outside their own environment while running GPU workloads — keeping the measurement environment under your own control is a real advantage.",[11,204,206],{"id":205},"summary","Summary",[16,208,209],{},"Unlike CPU and memory, GPU is an inflexible resource where requests must equal limits. Automation like VPA doesn't extend to GPU today, so estimation accuracy still depends on measurement. By grounding your approach in statistical concepts like P50\u002FP99, continuously monitoring with Prometheus + Grafana, and combining that with queue management tools like Kueue, you can avoid the \"job submitted at night, gone by morning\" incident.",[16,211,212,213,217],{},"As organizations run AI agents and AI workloads in production, infrastructure that supports this kind of grounded, measurement-based resource design only becomes more important. ",[45,214,216],{"href":169,"rel":215},[49],"Kubo"," lets you get started right away on GPU workload resource design, with monitoring built in standard on a lightweight K3s-based setup. Let's support the infrastructure of the AI era with measurement, not assumptions.",{"title":219,"searchDepth":220,"depth":220,"links":221},"",2,[222,223,224,225,226],{"id":13,"depth":220,"text":14},{"id":29,"depth":220,"text":30},{"id":102,"depth":220,"text":103},{"id":143,"depth":220,"text":144},{"id":205,"depth":220,"text":206},"2026-09-20","Why AI training jobs on K3s get OOMKilled from GPU memory shortages, why Vertical Pod Autoscaler doesn't cover GPU resources, and how to size requests\u002Flimits correctly using measurement-based design.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation",{"title":5,"description":228},"blog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation",[238,239,240,241,242,243,244],"k3s","kubernetes","gpu","oomkilled","vpa","ai-infrastructure","resource-management","8njfMkLlXlxyCQ65wwHbmF75TrBB8GEy84EijHVhWbw",[247,255,262,269,277,286],{"path":248,"title":249,"description":250,"date":251,"tags":252},"\u002Fblog\u002Fen\u002Fk3s-vpa-inplaceorrecreate-pod-resize-cgroup","VPA InPlaceOrRecreate Didn't Stop the Restart. Testing InPlacePodVerticalScaling on K3s","Does VPA's InPlaceOrRecreate mode and In-Place Pod Resize really avoid Pod restarts? We test QoS class constraints and the cgroup v1 trap on K3s with real configs and kubectl commands.","2026-10-17",[238,239,242,253,254],"autoscaling","cgroup",{"path":256,"title":257,"description":258,"date":259,"tags":260},"\u002Fblog\u002Fen\u002Fkubernetes-hpa-vpa-autoscaling-conflict-design","\"Running Both Feels Safer\" Is the Trap. Why Combining HPA and VPA in Kubernetes Can Send Autoscaling Into a Spiral","Enabling both HPA and VPA in Kubernetes can make two well-intentioned automations fight each other, sending autoscaling into an endless spiral. Here's why the conflict happens, and how to combine them safely with updateMode and metric separation.","2026-10-07",[238,239,261,242,253],"hpa",{"path":263,"title":264,"description":265,"date":266,"tags":267},"\u002Fblog\u002Fen\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral","Your HPA Config Is Correct. So Why Won't Pods Scale? The Real Cause of 'Unknown Metrics' in K3s and the HPA\u002FVPA Death Spiral","You set up an HPA in K3s and Pods still won't scale. The usual culprits are unset resource requests silently disabling metrics, and a self-reinforcing HPA\u002FVPA feedback loop. Here's how to diagnose and fix it.","2026-09-02",[238,239,261,242,253,268],"metrics-server",{"path":270,"title":271,"description":272,"date":273,"tags":274},"\u002Fblog\u002Fen\u002Fai-generated-kubernetes-manifest-resource-overprovisioning","Kubernetes Resource Design Can't Be Left to AI: Why 'Working' YAML Is Wasting 69% of Your Cloud Bill","AI-generated Kubernetes manifests pass kubectl apply and 'work' — but getting Kubernetes resource design wrong drives massive overprovisioning. Here's why AI struggles with production-grade requests\u002Flimits and what to check before you ship.","2026-08-04",[238,239,244,275,276],"capacity-planning","managed-kubernetes",{"path":278,"title":279,"description":280,"date":281,"tags":282},"\u002Fblog\u002Fen\u002Fkubernetes-mlops-gpu-scheduling-talent-shortage","In the Age of AI-Written Code, Why Are Infrastructure Engineers Getting Raises? Inside the 'MLOps Talent Shortage' Fueled by the Corporate AI Adoption Rush","Generative AI has made it possible for almost anyone to write code, yet as more companies adopt AI, demand for MLOps talent who can reliably run GPUs and model serving on Kubernetes keeps rising. Here's why, and how a managed Kubernetes platform can help.","2026-07-30",[239,238,283,243,284,285],"mlops","gpu-scheduling","devops",{"path":287,"title":288,"description":289,"date":290,"tags":291},"\u002Fblog\u002Fen\u002Fai-manifest-generation-kubernetes-architecture-bottleneck","AI Can Write a YAML File in One Second, But Your Cluster Won't Get Any Faster. Why the Real Bottleneck in Kubernetes Operations Is Architecture, Not Code","Generating Kubernetes manifests at AI speed doesn't make production Kubernetes operations faster. This article breaks down three real bottlenecks — CPU throttling, the HPA\u002FVPA conflict, and database connection starvation — and how to split the work between AI and humans.","2026-07-18",[239,238,244,253,292,285],"ai-ops",1792592634669]