[{"data":1,"prerenderedAt":495},["ShallowReactive",2],{"blog-en-k3s-edge-ai-inference-fractional-gpu-time-slicing":3,"blog-related-en-k3s-edge-ai-inference-fractional-gpu-time-slicing":435,"blog-en-k3s-edge-ai-inference-fractional-gpu-time-slicing-alt":423},{"id":4,"title":5,"author":6,"body":7,"date":417,"description":418,"extension":419,"image":420,"locale":421,"meta":422,"navigation":423,"path":424,"seo":425,"stem":426,"tags":427,"__hash__":434},"blog\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing.md","We Built an Edge AI Inference Stack on K3s, and Couldn't Fit Two Models on One GPU. The Fractional GPU Wall","Kubo Team",{"type":8,"value":9,"toc":409},"minimark",[10,15,23,31,39,50,53,57,63,66,79,85,92,101,106,109,113,119,136,179,199,206,210,216,223,231,236,243,256,261,264,268,274,283,315,318,342,351,354,358,364,367,370,381,384,397,405],[11,12,14],"h2",{"id":13},"_1-we-tried-to-add-a-second-inference-model-to-one-jetson-and-the-pod-never-scheduled","1. We Tried to Add a Second Inference Model to One Jetson, and the Pod Never Scheduled",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Fsection01.webp",[16,24,25,26,30],{},"Here's a story that plays out often for teams running edge AI inference on K3s. A single edge node has an image recognition model deployed and running fine. Then someone tries to deploy a second, different inference model to the same node — and the new Pod gets stuck in ",[27,28,29],"code",{},"Pending"," and never starts.",[16,32,33,34,38],{},"Checking the node's resources shows plenty of headroom on both CPU and memory. Yet the scheduler decides there isn't enough GPU and refuses to place the new Pod. This isn't a misconfiguration — it's ",[35,36,37],"strong",{},"a direct consequence of how standard Kubernetes manages GPU resources",".",[16,40,41,42,49],{},"This constraint applies equally whether you're running standard Kubernetes or K3s. If anything, it's the first wall you hit when running a lightweight distribution like K3s on a one-node-one-GPU edge setup. K3s runs with a minimum of 2GB RAM for the server and 512MB for the agent (",[43,44,48],"a",{"href":45,"rel":46},"https:\u002F\u002Fdocs.k3s.io\u002Finstallation\u002Frequirements",[47],"nofollow","K3s official documentation","), shipping as a single binary you can deploy in minutes — but that resource efficiency and the awkwardness of GPU as a hardware resource are two separate problems you have to deal with on their own terms.",[16,51,52],{},"This article digs into why this wall exists, starting from Kubernetes' design philosophy, and explains how to work around it when building an edge AI inference stack on K3s.",[11,54,56],{"id":55},"_2-the-root-cause-is-kubernetes-design-gpus-only-track-allocated-or-not-never-in-use","2. The Root Cause Is Kubernetes' Design: GPUs Only Track \"Allocated or Not,\" Never \"In Use\"",[16,58,59],{},[19,60],{"alt":61,"src":62},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Fsection02.webp",[16,64,65],{},"The first thing to understand is that Kubernetes treats GPUs fundamentally differently from CPU and memory.",[16,67,68,69,72,73,78],{},"CPU and memory are finely controlled by Linux kernel cgroups. You can request fractional units like ",[27,70,71],{},"500m"," (half a core), and whatever isn't used gets shared with other Pods. GPUs are a different story. The ",[43,74,77],{"href":75,"rel":76},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fadminister-cluster\u002Fextended-resource-node\u002F",[47],"Kubernetes official documentation"," states it plainly:",[80,81,82],"blockquote",{},[16,83,84],{},"Extended resources must be advertised in integer amounts. For example, a Node can advertise four dongles, but not 4.5 dongles.",[16,86,87,88,91],{},"In Kubernetes, a GPU is treated as an \"Extended Resource,\" which ",[35,89,90],{},"can only be advertised and allocated in whole-number units",". You can't request half a GPU — a Pod either claims an entire physical GPU, or none at all.",[16,93,94,95,100],{},"The reason traces back to the fact that GPUs sit outside cgroups' control. According to ",[43,96,99],{"href":97,"rel":98},"https:\u002F\u002Fscaleops.com\u002Fblog\u002Fgpus-in-kubernetes-how-it-actually-works-under-the-hood\u002F",[47],"ScaleOps' explanation",", while CPU and memory can be metered and throttled by kernel primitives, GPUs are \"opaque\" — an independent domain controlled by the NVIDIA driver. As a result, all the Kubernetes scheduler can see is whether a GPU has been allocated at all:",[80,102,103],{},[16,104,105],{},"The resource count tracks allocation, not use.",[16,107,108],{},"Actual utilization is completely invisible to the scheduler. A Pod using 0% of a GPU and one using 99% both look identical: \"one GPU occupied.\" This is the root cause of why you can't simply co-locate a second inference model on the same GPU.",[11,110,112],{"id":111},"_3-how-the-nvidia-device-plugin-and-time-slicing-work-turning-one-gpu-into-several-on-paper","3. How the NVIDIA Device Plugin and Time-Slicing Work: Turning One GPU Into \"Several\" on Paper",[16,114,115],{},[19,116],{"alt":117,"src":118},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Fsection03.webp",[16,120,121,122,125,126,131,132,135],{},"Kubernetes doesn't even know a GPU exists by default. To make GPUs assignable to Pods, you need to deploy the ",[35,123,124],{},"NVIDIA device plugin"," as a DaemonSet on every node. According to the ",[43,127,130],{"href":128,"rel":129},"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fk8s-device-plugin",[47],"official NVIDIA device plugin repository",", this plugin auto-detects the number of GPUs on each node and registers them with the kubelet under the resource name ",[27,133,134],{},"nvidia.com\u002Fgpu",". Pods then request the resource like this:",[137,138,143],"pre",{"className":139,"code":140,"language":141,"meta":142,"style":142},"language-yaml shiki shiki-themes tokyo-night","resources:\n  limits:\n    nvidia.com\u002Fgpu: 1\n","yaml","",[27,144,145,158,166],{"__ignoreMap":142},[146,147,150,154],"span",{"class":148,"line":149},"line",1,[146,151,153],{"class":152},"s0U2E","resources",[146,155,157],{"class":156},"sAklC",":\n",[146,159,161,164],{"class":148,"line":160},2,[146,162,163],{"class":152},"  limits",[146,165,157],{"class":156},[146,167,169,172,175],{"class":148,"line":168},3,[146,170,171],{"class":152},"    nvidia.com\u002Fgpu",[146,173,174],{"class":156},":",[146,176,178],{"class":177},"sOJ5S"," 1\n",[16,180,181,182,185,186,191,192,195,196,198],{},"This is where ",[35,183,184],{},"time-slicing"," comes in. According to the ",[43,187,190],{"href":188,"rel":189},"https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002F26.3\u002Fgpu-sharing.html",[47],"NVIDIA official documentation",", time-slicing lets you advertise a single physical GPU as multiple \"replicas\" by setting ",[27,193,194],{},"resources.replicas"," in a ConfigMap. Configure 4 replicas on a GPU, for instance, and a node with 2 physical GPUs will advertise 8 ",[27,197,134],{}," resources in total. To the Kubernetes scheduler, it now looks like there are 8 GPUs available, letting multiple Pods share a single physical GPU.",[16,200,201,202,205],{},"There's a caveat, though. The same documentation explicitly notes that \"requests to multiple time-sliced GPUs do not guarantee any level of proportional compute allocation.\" Time-slicing simply ",[35,203,204],{},"multiplexes workloads in time","; it doesn't mean every Pod gets full throughput simultaneously. If inference latency matters to your SLA, you need to design around this characteristic.",[11,207,209],{"id":208},"_4-the-edge-specific-trap-no-mig-and-jetsons-unified-memory","4. The Edge-Specific Trap: No MIG, and Jetson's Unified Memory",[16,211,212],{},[19,213],{"alt":214,"src":215},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Fsection04.webp",[16,217,218,219,222],{},"Besides time-slicing, there's another way to share a GPU: ",[35,220,221],{},"MIG (Multi-Instance GPU)",", a hardware-level isolation feature. MIG splits a physical GPU into multiple independent partitions with fully isolated memory and fault domains.",[16,224,225,226,174],{},"But MIG comes with a significant catch. According to ",[43,227,230],{"href":228,"rel":229},"https:\u002F\u002Fwww.vcluster.com\u002Fblog\u002Fdiy-gpu-sharing-in-kubernetes",[47],"vCluster's explainer",[80,232,233],{},[16,234,235],{},"MIG's only supported on select NVIDIA GPUs, starting from the Ampere generation introduced in 2020.",[16,237,238,239,242],{},"MIG is only available on ",[35,240,241],{},"high-end datacenter GPUs"," from the Ampere generation (2020) onward. Edge SoCs like NVIDIA Jetson don't support it at all. That means on the edge, time-slicing is effectively your only option for sharing a GPU. Because time-slicing isolates neither memory nor faults, a single runaway Pod can affect every other inference workload sharing that GPU. If you design multi-model co-location at the edge without understanding this reliability gap, you can end up unable to isolate the root cause when a production incident hits.",[16,244,245,246,251,252,255],{},"Jetson-class edge devices have another structural difference worth noting. In a datacenter GPU server, CPU memory and GPU VRAM are physically separate, and tensors are copied across PCIe. According to ",[43,247,250],{"href":248,"rel":249},"https:\u002F\u002Fiotdigitaltwinplm.com\u002Fnvidia-jetson-k3s-edge-ai-cluster-tutorial-2026\u002F",[47],"iotdigitaltwinplm's explainer",", Jetson instead uses a ",[35,253,254],{},"Unified Memory"," architecture, where the CPU and GPU share a single memory pool.",[80,257,258],{},[16,259,260],{},"Size your inference pod's memory request against the whole board, not against an imaginary separate GPU.",[16,262,263],{},"In other words, the cloud-GPU-server assumption of \"we have 80GB of VRAM, so we're fine\" simply doesn't hold on Jetson. The model, its activations, the OS, and K3s itself are all competing for the same memory pool — so unless you size Pod resources against the whole board's memory, you'll run out of memory far more easily than you'd expect.",[11,265,267],{"id":266},"_5-implementing-it-on-k3s-time-slicing-configuration-and-resource-sizing","5. Implementing It on K3s: Time-Slicing Configuration and Resource Sizing",[16,269,270],{},[19,271],{"alt":272,"src":273},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Fsection05.webp",[16,275,276,277,282],{},"Drawing on ",[43,278,281],{"href":279,"rel":280},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-20-k3s-edge-ai-ml\u002Fview",[47],"oneuptime's hands-on guide",", the following settings are the bare minimum for building an edge AI inference stack on K3s.",[284,285,286,296,309],"ul",{},[287,288,289,295],"li",{},[35,290,291,292],{},"Set ",[27,293,294],{},"runtimeClassName: nvidia"," — containerd needs templated configuration so it defaults to the NVIDIA runtime",[287,297,298,304,305,308],{},[35,299,300,301],{},"Use a ",[27,302,303],{},"nodeSelector"," to explicitly target GPU nodes (e.g. ",[27,306,307],{},"ai-capable: \"true\"",") — this prevents accidental scheduling onto CPU-only nodes",[287,310,311,314],{},[35,312,313],{},"Set realistic memory requests\u002Flimits"," — the guide's example uses a request of 2–3Gi and a limit of 4–6Gi, sized with headroom against the whole board's memory",[16,316,317],{},"When GPU scheduling doesn't behave as expected, there are three typical failure patterns.",[319,320,321,330,336],"ol",{},[287,322,323,326,327,329],{},[35,324,325],{},"Missing device plugin",": The Pod stays ",[27,328,29],{}," with an insufficient-resource error",[287,331,332,335],{},[35,333,334],{},"Wrong runtime configuration",": The Pod gets scheduled but can't access CUDA, silently falling back to CPU — a performance hit that's easy to miss from logs alone",[287,337,338,341],{},[35,339,340],{},"Missing RuntimeClass",": Resources are allocated, but the GPU device never actually gets mounted into the container",[16,343,344,345,350],{},"All three show up as \"the Pod looks like it's running, but performance isn't there,\" so you need latency and throughput monitoring running alongside the deployment to catch them. Being able to see, on a dashboard, exactly which Pod is holding which GPU allocation makes this kind of silent degradation much easier to catch early. ",[43,346,349],{"href":347,"rel":348},"https:\u002F\u002Fkubo.hexabase.io\u002F",[47],"Kubo","'s Captain UI gives you exactly that visibility into cluster resource allocation, instead of leaving it a black box.",[16,352,353],{},"If you want to co-locate multiple models, configure time-slicing replicas via the NVIDIA GPU Operator's ConfigMap, and verify up front whether each inference workload's latency requirements can coexist with time-slicing's \"no proportional allocation guarantee.\" Keeping SLA-sensitive inference separate from batch-tolerant inference on different GPUs is a practical way to hedge this risk.",[11,355,357],{"id":356},"_6-conclusion-if-you-choose-k3s-for-edge-ai-inference-design-around-the-limits-of-gpu-sharing","6. Conclusion: If You Choose K3s for Edge AI Inference, Design Around the Limits of GPU Sharing",[16,359,360],{},[19,361],{"alt":362,"src":363},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Fsection06.webp",[16,365,366],{},"K3s's appeal is its ability to run with a minimum of 2GB RAM on the server and 512MB on the agent, which has made deploying Kubernetes to the edge practical. But the constraints of GPU as a physical resource are a Kubernetes-wide specification, not something K3s can sidestep just by being lightweight.",[16,368,369],{},"To summarize, there are three points to keep in mind when designing an edge AI inference stack:",[284,371,372,375,378],{},[287,373,374],{},"GPUs are allocated in whole-number units with request=limit fixed; fractional allocation isn't possible out of the box",[287,376,377],{},"Co-locating multiple models on one GPU requires time-slicing, which isolates neither memory nor faults. Hardware-level isolation via MIG is limited to datacenter GPUs (Ampere generation or later) and isn't available at the edge",[287,379,380],{},"Edge devices like Jetson use a unified memory architecture, so you need to size against the whole board's memory rather than assume cloud-GPU-server-style resource design",[16,382,383],{},"The move to deploy Kubernetes beyond the cloud — into factories, retail stores, and other edge environments — keeps gaining ground. K3s's lightweight footprint is what makes edge AI inference achievable at a realistic cost. But if you design it with \"the same mindset as the cloud\" without understanding the limits of GPU sharing, you'll run into invisible performance degradation and hard-to-diagnose incidents in production.",[16,385,386,387,390,391,396],{},"Working out these edge-AI-specific design decisions from scratch every time you build is a heavy lift. ",[43,388,349],{"href":347,"rel":389},[47],", a managed Kubernetes service built on K3s, lets you configure GPU nodes and use AI-Driven Deployment to build this kind of setup without falling into YAML hell. And if you want to run AI agents on top of your inference infrastructure, ",[43,392,395],{"href":393,"rel":394},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fcaptain-ai\u002F",[47],"Captain.AI"," integrates as the execution layer for those AI workloads.",[16,398,399,400,38],{},"If you'd like to talk through the design of your edge AI inference stack, feel free to ",[43,401,404],{"href":402,"rel":403},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[47],"get in touch",[406,407,408],"style",{},"html pre.shiki code .s0U2E, html code.shiki .s0U2E{--shiki-default:#F7768E}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sOJ5S, html code.shiki .sOJ5S{--shiki-default:#FF9E64}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":142,"searchDepth":160,"depth":160,"links":410},[411,412,413,414,415,416],{"id":13,"depth":160,"text":14},{"id":55,"depth":160,"text":56},{"id":111,"depth":160,"text":112},{"id":208,"depth":160,"text":209},{"id":266,"depth":160,"text":267},{"id":356,"depth":160,"text":357},"2026-10-02","Building edge AI inference on K3s runs into standard Kubernetes GPU resource management (integer units, request=limit). Here's how to share one GPU across multiple models, and what to watch for with time-slicing.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing",{"title":5,"description":418},"blog\u002Fen\u002Fk3s-edge-ai-inference-fractional-gpu-time-slicing",[428,429,430,431,432,184,433],"k3s","kubernetes","edge-ai","gpu-scheduling","device-plugin","fractional-gpu","Pr25xi14NfDmUy8rhmgZrr-ZeNHTazF50kF8ikVQviU",[436,445,453,463,475,485],{"path":437,"title":438,"description":439,"date":440,"tags":441},"\u002Fblog\u002Fen\u002Fk3s-ai-inference-gpu-scheduling-dra-keda-kserve","Kubernetes Hasn't Changed Its Design in 10 Years — Here Are 3 Assumptions Your K3s Cluster Must Rethink for AI Inference Workloads","Ten years after Kubernetes launched, AI agents and LLM inference have upended three of its core assumptions: how GPUs are allocated, how autoscaling decisions are made, and how model serving is built. Here's how to rethink your K3s cluster with Dynamic Resource Allocation, KEDA, and KServe.","2026-08-31",[428,429,442,431,443,444],"ai-workload","dynamic-resource-allocation","mlops",{"path":446,"title":447,"description":448,"date":449,"tags":450},"\u002Fblog\u002Fen\u002Fkubernetes-mlops-gpu-scheduling-talent-shortage","In the Age of AI-Written Code, Why Are Infrastructure Engineers Getting Raises? Inside the 'MLOps Talent Shortage' Fueled by the Corporate AI Adoption Rush","Generative AI has made it possible for almost anyone to write code, yet as more companies adopt AI, demand for MLOps talent who can reliably run GPUs and model serving on Kubernetes keeps rising. Here's why, and how a managed Kubernetes platform can help.","2026-07-30",[429,428,444,451,431,452],"ai-infrastructure","devops",{"path":454,"title":455,"description":456,"date":457,"tags":458},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[428,429,459,460,461,462],"hpa","containerresource","autoscaling","service-mesh",{"path":464,"title":465,"description":466,"date":467,"tags":468},"\u002Fblog\u002Fen\u002Fk3s-vs-kubernetes-cni-flannel-default-comparison","K3s vs Standard Kubernetes: How Different Their CNI Starting Points Really Are","K3s ships with Flannel built in, while standard Kubernetes starts with no CNI selected at all. This comparison, grounded in the official K3s, Kubernetes, Calico, and Cilium docs, explains how that difference affects NetworkPolicy enforcement and network performance.","2026-10-20",[428,429,469,470,471,472,473,474],"cni","flannel","calico","cilium","networkpolicy","networking",{"path":476,"title":477,"description":478,"date":479,"tags":480},"\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending","We Drained a Node and a Pod Never Came Back. Why Topology Spread Constraints and PodDisruptionBudget Collide on K3s","Draining a node in production K3s left a Pod stuck Pending. The cause: Topology Spread Constraints' maxSkew colliding with PodDisruptionBudget. We verify how draining itself creates a node shortage, based on official documentation.","2026-10-19",[428,429,481,482,483,484],"topology-spread-constraints","poddisruptionbudget","scheduling","node-drain",{"path":486,"title":487,"description":488,"date":489,"tags":490},"\u002Fblog\u002Fen\u002Fkyverno-validatingadmissionpolicy-cel-migration-silent-bypass","Is the CEL Rewrite of Your Policy Really the Same? How Migrating from Kyverno to ValidatingAdmissionPolicy Let Some Resources Silently Slip Through","As Kyverno's ClusterPolicy heads toward deprecation, teams migrating to ValidatingAdmissionPolicy (CEL) have found that validation silently skipped for certain resource types. Here's the CEL runtime error and failurePolicy trap behind it, and a safe migration path.","2026-10-18",[428,429,491,492,493,494],"validatingadmissionpolicy","kyverno","cel","admission-control",1792592633560]