A Job Submitted at Night Vanished Without a Trace by Morning
You submit a GPU fine-tuning job to a shared Kubernetes cluster and check on it the next morning, only to find it was OOMKilled (forcibly terminated due to insufficient memory). Plenty of operators have hit this exact pattern. The log shows nothing but a single "Killed" line, and hours of compute cost are gone.
What makes this tricky is that the usual fix for CPU or memory pods — "just bump up the requests a bit" — doesn't work for GPUs. GPUs are treated specially within Kubernetes, and if you design your resource allocation without understanding that special treatment, you'll repeat the same incident.
Why "Padding Your Requests" Doesn't Solve It
CPU and memory requests/limits can be set relatively flexibly. CPU is throttled gently, and memory can absorb some variation if you set requests with a bit of headroom.
GPUs are a different story. The official Kubernetes documentation spells out the rules for specifying GPU resources:
- You can set GPU
limitswithoutrequests— Kubernetes will use the limit value as the request by default - If you specify both
limitsandrequests, the two values must be equal - You cannot specify GPU
requestswithout also specifyinglimits
In other words, GPUs have none of the "keep requests low, limits high" buffering that CPU allows. Whatever GPU a Pod requests is exclusively occupied, and any slack isn't shared with other Pods. That's exactly why a single miscalculation in your requests estimate translates directly into an OOMKilled event or a scheduling failure.
On top of that, the Fairwinds blog notes that "fleet average GPU utilization is often well below 30 percent, with recent data putting it as low as 5 percent." In many production environments, GPU resource sizing gets decided once at deploy time and is never revisited.
The Inconvenient Truth: You Can't Hand This Off to VPA
"Just let autoscaling handle it" doesn't hold up for GPU workloads.
Check the official Kubernetes Vertical Pod Autoscaler documentation and you'll find the Recommender consistently analyzes "CPU and memory" — GPU is never mentioned. Google Cloud's GKE documentation states the same scope: "by default, vertical Pod autoscaling computes recommendations for both CPU and memory." ScaleOps's VPA explainer likewise frames the resource parameters VPA manages — requests and limits — consistently around CPU and memory.
Teams that have automated their operations with HPA and VPA are especially prone to assuming "GPU will get auto-optimized the same way." But GPU resource sizing remains, for now, a domain that still requires humans to measure and design manually.
So What Should Your Estimates Be Based On? A Measurement-Driven Design Process
If you can't rely on automation, you have to rely on measured data. OneUptime's resource sizing guide suggests setting CPU requests to the median (P50) and memory requests to the P99 of actual measured usage, with memory limits set at 1.5-2x requests. Since GPU forces requests to equal limits, the "give limits some headroom" approach doesn't directly apply — but the underlying sequence, measure first, then decide the numbers, still carries over.
For the measurement layer, a setup that continuously visualizes GPU utilization at the node and Pod level with Prometheus + Grafana is the foundation. Kubo Cloud bundles this Prometheus + Grafana monitoring standard into its K3s-based setup, so you can start measuring GPU usage without building a separate monitoring stack.
As operations mature, the challenge shifts from sizing alone to allocation — who gets how much GPU. Kueue, a CNCF-hosted project, provides quota management and job queueing for batch and AI/ML workloads, and can dynamically borrow capacity from idle resource pools. A CNCF blog post from August 2026 describes an "AI factory" operating model where multiple teams safely share the same GPU pool, with a combination of technologies like DRA and Kueue being key to the implementation. The mechanism that exposes GPUs to Nodes and Pods in the first place is handled by NVIDIA's official device plugin — understanding these layers before designing your requests/limits is the shortest path to preventing OOMKilled incidents.
If you're planning for large-scale multi-tenant operations, it's worth considering a platform like Kubo On-Premise, where you can measure and tune GPU resources incrementally on your own infrastructure. This matters especially for industries that can't send data outside their own environment while running GPU workloads — keeping the measurement environment under your own control is a real advantage.
Summary
Unlike CPU and memory, GPU is an inflexible resource where requests must equal limits. Automation like VPA doesn't extend to GPU today, so estimation accuracy still depends on measurement. By grounding your approach in statistical concepts like P50/P99, continuously monitoring with Prometheus + Grafana, and combining that with queue management tools like Kueue, you can avoid the "job submitted at night, gone by morning" incident.
As organizations run AI agents and AI workloads in production, infrastructure that supports this kind of grounded, measurement-based resource design only becomes more important. Kubo lets you get started right away on GPU workload resource design, with monitoring built in standard on a lightweight K3s-based setup. Let's support the infrastructure of the AI era with measurement, not assumptions.