AI Compute Demand Was Built Around Training. That's Quietly Reversing.
Chances are you've noticed the phrase "Kubernetes AI inference" showing up more often over the past year. At the keynote of KubeCon + CloudNativeCon Japan 2026, CNCF leadership spoke about a shift in how AI computing capacity is being allocated. Until a few years ago, the vast majority of AI-related compute went toward model "training." Today, the center of gravity is moving to "inference."
According to an analysis of data center demand, AI inference compute capacity is projected to exceed 90 gigawatts by 2030, growing at a 35% CAGR. McKinsey's own estimates are even more specific: data center demand for AI training is expected to grow from 23.1 gigawatts in 2025 to 62.2 gigawatts by 2030 (22% CAGR), while demand for inference is expected to surge from 20.9 gigawatts to 93.3 gigawatts over the same period (35% CAGR). By 2030, inference is projected to account for more than half of all AI compute, overtaking training as the single largest workload category.
Other research points in the same direction. An article on data center growth trends cites Deloitte's estimate that inference already accounted for half of all AI compute in 2025 and will reach two-thirds by 2026, while Brookfield projects inference will make up 75% of AI compute demand by 2030. The exact numbers vary, but the direction is consistent: infrastructure designed around training no longer matches reality.
Given the scale of this shift, cost efficiency and freedom from vendor lock-in are becoming critical decision criteria for any organization choosing or re-evaluating its infrastructure. It's part of why managed K3s-based Kubernetes platforms like Kubo are increasingly being considered.
What makes this shift tricky is that training and inference demand fundamentally different things from infrastructure. The next section looks at how that difference is reshaping Kubernetes operations.
Why Inference Workloads Are Forcing Kubernetes to Rebuild Itself
Training jobs are essentially batch processes: they consume a large pool of GPUs for hours or days at a stretch, then release them. Inference workloads, on the other hand, run continuously, scale up and down within seconds or minutes based on request volume, and translate any latency degradation directly into a worse user experience. Scheduling and resource-management approaches optimized for training tend to waste resources when applied directly to inference.
Kubernetes itself is adding inference-oriented capabilities to address this. A prime example is "In-Place Pod Resize." According to the official Kubernetes blog, this feature lets you change a container's CPU and memory requests/limits without restarting the Pod, and it graduated to beta with default-enabled status in v1.33. Inference serving often needs a burst of CPU right after startup and then settles into a lower steady state — being able to adjust resources without a restart has a direct impact on availability.
The Google Cloud blog further highlights four technical building blocks for supporting both inference and training: "Dynamic Resource Allocation" (DRA) for fine-grained control over GPUs/TPUs, "All-or-Nothing Scheduling" so distributed training jobs wait until all required resources are available, custom-metrics-based autoscaling driven by signals like GPU utilization, and standardized observability for accelerators. Together, these show Kubernetes expanding its role from "container orchestrator" to "AI workload platform."
If every cloud and distribution implements these extensions independently, the result is a patchwork of environments where things only "just work" in specific places. That's exactly the fragmentation problem CNCF's standardization effort is trying to solve.
What Is the CNCF's "AI Conformance Program" Actually Guaranteeing?
According to CNCF's official announcement, the Certified Kubernetes AI Conformance Program was first announced in beta at KubeCon Japan in June 2025 and officially launched at KubeCon North America in Atlanta on November 11 of the same year. The goal is straightforward: ensure interoperability and portability for AI workloads and reduce fragmentation across vendors. Major cloud providers — AWS, Google Cloud, Microsoft Azure, Oracle, Red Hat, and others — have already joined as certified platforms.
A Forbes explainer notes that the program emerged in response to research showing that 82% of organizations are building custom AI solutions, with 58% of them relying on Kubernetes — and that incompatibility between platforms was raising the risk of vendor lock-in. With certification in place, an AI application validated on one Kubernetes environment can be expected to behave the same way on any other certified environment.
The program's requirements are also becoming stricter over time. According to reporting from Cloud Native Now, the 2026 KubeCon Europe event introduced the "Kubernetes AI Requirements (KAR) v1.35" certification standard, adding stable support for In-Place Pod Resize and Workload-Aware Scheduling as mandatory requirements. At the same time, new platforms — including OVHcloud, SpectroCloud, JD Cloud, and China Unicom Cloud — achieved certification, bringing the total number of certified platforms to 31. CNCF's Jonathan Bryce has said that "AI inference workloads will, in the near future, come to overwhelm every other class of workload running on Kubernetes clusters," and the detailed requirements are being developed in the open on GitHub.
In short, what CNCF is aiming for is a state where "even as inference workloads surge, you get consistent behavior regardless of environment." That same principle turns out to be a useful lens for choosing a managed Kubernetes platform.
What Should Managed K3s Environments Do to Meet Inference-Era Infrastructure Requirements?
Pulling together everything covered so far, the requirements for inference-era infrastructure boil down to three things. First, autoscaling based on AI-specific signals like GPU utilization and inference latency. Second, observability that surfaces accelerator metrics in a standardized way. Third, portability that isn't tied to a specific cloud or distribution — in other words, avoiding vendor lock-in.
These three conditions are exactly where using a fully managed service like EKS or AKS on its own tends to force a trade-off between cost and lock-in. Kubo, a K3s-based managed Kubernetes platform, is built on standards-compliant Pure Kubernetes, which aligns naturally with the environment-level consistency CNCF's certification is aiming for. Its built-in Prometheus + Grafana stack also gives you a ready-made foundation for visualizing accelerator metrics.
Much of today's inference workload runs as part of AI agents or MLOps pipelines. For teams thinking ahead to operating these autonomous workloads, combining Kubo with an AI agent execution platform like Captain.AI is one option worth considering. Built-in GitOps and Helm chart support also means you don't need to rebuild your inference deployment pipeline from scratch.
Summary
AI compute demand is flipping from "training-centric" to "inference-centric," and multiple research organizations now project that inference compute capacity will substantially outpace training by 2030. Kubernetes is responding by adopting inference-specific capabilities like In-Place Pod Resize and Dynamic Resource Allocation, while CNCF's AI Conformance Program pushes toward a standard where workloads "behave the same regardless of environment."
Given this trajectory, cluster design going forward needs to prioritize three things: autoscaling based on inference-specific metrics, observability, and portability that doesn't lock you into a single vendor. For organizations without the resources to build robust infrastructure from the ground up, Kubo — a standards-compliant, cost-efficient, K3s-based platform — is a realistic option for supporting inference-era workloads. Start by getting in touch to map out where your inference infrastructure stands today.