We Built an Edge AI Inference Stack on K3s, and Couldn't Fit Two Models on One GPU. The Fractional GPU Wall

1. We Tried to Add a Second Inference Model to One Jetson, and the Pod Never Scheduled

Here's a story that plays out often for teams running edge AI inference on K3s. A single edge node has an image recognition model deployed and running fine. Then someone tries to deploy a second, different inference model to the same node — and the new Pod gets stuck in Pending and never starts.

Checking the node's resources shows plenty of headroom on both CPU and memory. Yet the scheduler decides there isn't enough GPU and refuses to place the new Pod. This isn't a misconfiguration — it's a direct consequence of how standard Kubernetes manages GPU resources.

This constraint applies equally whether you're running standard Kubernetes or K3s. If anything, it's the first wall you hit when running a lightweight distribution like K3s on a one-node-one-GPU edge setup. K3s runs with a minimum of 2GB RAM for the server and 512MB for the agent (K3s official documentation), shipping as a single binary you can deploy in minutes — but that resource efficiency and the awkwardness of GPU as a hardware resource are two separate problems you have to deal with on their own terms.

This article digs into why this wall exists, starting from Kubernetes' design philosophy, and explains how to work around it when building an edge AI inference stack on K3s.

2. The Root Cause Is Kubernetes' Design: GPUs Only Track "Allocated or Not," Never "In Use"

The first thing to understand is that Kubernetes treats GPUs fundamentally differently from CPU and memory.

CPU and memory are finely controlled by Linux kernel cgroups. You can request fractional units like 500m (half a core), and whatever isn't used gets shared with other Pods. GPUs are a different story. The Kubernetes official documentation states it plainly:

Extended resources must be advertised in integer amounts. For example, a Node can advertise four dongles, but not 4.5 dongles.

In Kubernetes, a GPU is treated as an "Extended Resource," which can only be advertised and allocated in whole-number units. You can't request half a GPU — a Pod either claims an entire physical GPU, or none at all.

The reason traces back to the fact that GPUs sit outside cgroups' control. According to ScaleOps' explanation, while CPU and memory can be metered and throttled by kernel primitives, GPUs are "opaque" — an independent domain controlled by the NVIDIA driver. As a result, all the Kubernetes scheduler can see is whether a GPU has been allocated at all:

The resource count tracks allocation, not use.

Actual utilization is completely invisible to the scheduler. A Pod using 0% of a GPU and one using 99% both look identical: "one GPU occupied." This is the root cause of why you can't simply co-locate a second inference model on the same GPU.

3. How the NVIDIA Device Plugin and Time-Slicing Work: Turning One GPU Into "Several" on Paper

Kubernetes doesn't even know a GPU exists by default. To make GPUs assignable to Pods, you need to deploy the NVIDIA device plugin as a DaemonSet on every node. According to the official NVIDIA device plugin repository, this plugin auto-detects the number of GPUs on each node and registers them with the kubelet under the resource name nvidia.com/gpu. Pods then request the resource like this:

resources:
  limits:
    nvidia.com/gpu: 1

This is where time-slicing comes in. According to the NVIDIA official documentation, time-slicing lets you advertise a single physical GPU as multiple "replicas" by setting resources.replicas in a ConfigMap. Configure 4 replicas on a GPU, for instance, and a node with 2 physical GPUs will advertise 8 nvidia.com/gpu resources in total. To the Kubernetes scheduler, it now looks like there are 8 GPUs available, letting multiple Pods share a single physical GPU.

There's a caveat, though. The same documentation explicitly notes that "requests to multiple time-sliced GPUs do not guarantee any level of proportional compute allocation." Time-slicing simply multiplexes workloads in time; it doesn't mean every Pod gets full throughput simultaneously. If inference latency matters to your SLA, you need to design around this characteristic.

4. The Edge-Specific Trap: No MIG, and Jetson's Unified Memory

Besides time-slicing, there's another way to share a GPU: MIG (Multi-Instance GPU), a hardware-level isolation feature. MIG splits a physical GPU into multiple independent partitions with fully isolated memory and fault domains.

But MIG comes with a significant catch. According to vCluster's explainer:

MIG's only supported on select NVIDIA GPUs, starting from the Ampere generation introduced in 2020.

MIG is only available on high-end datacenter GPUs from the Ampere generation (2020) onward. Edge SoCs like NVIDIA Jetson don't support it at all. That means on the edge, time-slicing is effectively your only option for sharing a GPU. Because time-slicing isolates neither memory nor faults, a single runaway Pod can affect every other inference workload sharing that GPU. If you design multi-model co-location at the edge without understanding this reliability gap, you can end up unable to isolate the root cause when a production incident hits.

Jetson-class edge devices have another structural difference worth noting. In a datacenter GPU server, CPU memory and GPU VRAM are physically separate, and tensors are copied across PCIe. According to iotdigitaltwinplm's explainer, Jetson instead uses a Unified Memory architecture, where the CPU and GPU share a single memory pool.

Size your inference pod's memory request against the whole board, not against an imaginary separate GPU.

In other words, the cloud-GPU-server assumption of "we have 80GB of VRAM, so we're fine" simply doesn't hold on Jetson. The model, its activations, the OS, and K3s itself are all competing for the same memory pool — so unless you size Pod resources against the whole board's memory, you'll run out of memory far more easily than you'd expect.

5. Implementing It on K3s: Time-Slicing Configuration and Resource Sizing

Drawing on oneuptime's hands-on guide, the following settings are the bare minimum for building an edge AI inference stack on K3s.

• Set runtimeClassName: nvidia — containerd needs templated configuration so it defaults to the NVIDIA runtime
• Use a nodeSelector to explicitly target GPU nodes (e.g. ai-capable: "true") — this prevents accidental scheduling onto CPU-only nodes
• Set realistic memory requests/limits — the guide's example uses a request of 2–3Gi and a limit of 4–6Gi, sized with headroom against the whole board's memory

When GPU scheduling doesn't behave as expected, there are three typical failure patterns.

1. Missing device plugin: The Pod stays Pending with an insufficient-resource error
2. Wrong runtime configuration: The Pod gets scheduled but can't access CUDA, silently falling back to CPU — a performance hit that's easy to miss from logs alone
3. Missing RuntimeClass: Resources are allocated, but the GPU device never actually gets mounted into the container

All three show up as "the Pod looks like it's running, but performance isn't there," so you need latency and throughput monitoring running alongside the deployment to catch them. Being able to see, on a dashboard, exactly which Pod is holding which GPU allocation makes this kind of silent degradation much easier to catch early. Kubo's Captain UI gives you exactly that visibility into cluster resource allocation, instead of leaving it a black box.

If you want to co-locate multiple models, configure time-slicing replicas via the NVIDIA GPU Operator's ConfigMap, and verify up front whether each inference workload's latency requirements can coexist with time-slicing's "no proportional allocation guarantee." Keeping SLA-sensitive inference separate from batch-tolerant inference on different GPUs is a practical way to hedge this risk.

6. Conclusion: If You Choose K3s for Edge AI Inference, Design Around the Limits of GPU Sharing

K3s's appeal is its ability to run with a minimum of 2GB RAM on the server and 512MB on the agent, which has made deploying Kubernetes to the edge practical. But the constraints of GPU as a physical resource are a Kubernetes-wide specification, not something K3s can sidestep just by being lightweight.

To summarize, there are three points to keep in mind when designing an edge AI inference stack:

• GPUs are allocated in whole-number units with request=limit fixed; fractional allocation isn't possible out of the box
• Co-locating multiple models on one GPU requires time-slicing, which isolates neither memory nor faults. Hardware-level isolation via MIG is limited to datacenter GPUs (Ampere generation or later) and isn't available at the edge
• Edge devices like Jetson use a unified memory architecture, so you need to size against the whole board's memory rather than assume cloud-GPU-server-style resource design

The move to deploy Kubernetes beyond the cloud — into factories, retail stores, and other edge environments — keeps gaining ground. K3s's lightweight footprint is what makes edge AI inference achievable at a realistic cost. But if you design it with "the same mindset as the cloud" without understanding the limits of GPU sharing, you'll run into invisible performance degradation and hard-to-diagnose incidents in production.

Working out these edge-AI-specific design decisions from scratch every time you build is a heavy lift. Kubo, a managed Kubernetes service built on K3s, lets you configure GPU nodes and use AI-Driven Deployment to build this kind of setup without falling into YAML hell. And if you want to run AI agents on top of your inference infrastructure, Captain.AI integrates as the execution layer for those AI workloads.

If you'd like to talk through the design of your edge AI inference stack, feel free to get in touch.