K3s Pods Kept Getting Evicted. Disk Space Was Fine — Inodes Were Not

One morning, pods across several nodes in a K3s cluster suddenly started getting evicted, one after another. Running kubectl get pod -A showed a long list of pods marked Evicted. The first suspect is always disk space. But df -h showed plenty of free space on every node. Yet the nodes stayed in a DiskPressure state, refusing to schedule new pods.

This article explains a frequently overlooked relationship in K3s/Kubernetes operations: the one between kubelet's eviction manager and inode exhaustion, walking through a realistic troubleshooting scenario. If you only watch disk capacity in bytes, you'll never spot the real reason pods keep getting evicted on K3s. This kind of "invisible resource exhaustion" is exactly the sort of thing that turns on-call response into tribal knowledge. If you're evaluating a managed K3s environment like Kubo, the monitoring points covered here should give you a concrete checklist to judge it by.

1. df -h Won't Show the Culprit: Inode Exhaustion, the Limit on "File-Creation Capacity"

Linux filesystems maintain a separate accounting structure from disk capacity (bytes): the inode. An inode is a data structure that records a file's or directory's metadata — owner, permissions, the location of its data blocks — and one inode is consumed every time a file is created. Filesystems like ext4 allocate a fixed number of inodes at format time, based on a ratio (the inode_ratio, 16,384 bytes per inode by default), and that number cannot be changed later.

In other words, a workload that stores a few large files has plenty of headroom, but a workload that creates huge numbers of small files will run out of inodes long before it runs out of bytes. In research published in October 2026, the CNCF — the foundation behind cloud native technologies — documented a real cluster where disk usage sat at 83% and inode usage at 67%, caused by a Node.js node_modules directory (21,553 files per snapshot) duplicated across multiple container image layers (see the CNCF blog). Per that post's calculation, with a standard ext4 inode_ratio, inodes can run out at as little as 25% disk usage.

Why it's so easy to miss

• df -h only shows free capacity in bytes
• Most monitoring dashboards aggregate disk usage by percentage of bytes
• Processes that generate huge numbers of small files — logs, caches, temp files — rarely get a second thought

2. The Six Signals kubelet Watches, and the Early Warning That Doesn't Exist

Per the official Kubernetes documentation, kubelet monitors multiple eviction signals to detect node resource pressure (see Kubernetes docs: Node-pressure Eviction). For disk and inodes alone, there are six distinct signals.

According to a guide from the cloud native monitoring vendor Netdata, the default Linux hard eviction thresholds are nodefs: available capacity below 10%, inodes free below 5%, and imagefs: available capacity below 15%, inodes free below 5% (see Netdata: Kubernetes node DiskPressure).

There's early warning for bytes, but none for inodes

This is where a structural gap shows up. kubelet's image garbage collection kicks in once imagefs usage hits a byte-based high watermark of 85%, cleaning up unused images down to a low watermark of 80% — a graduated, byte-based early intervention. But for inodes, the only monitored point is the hard eviction threshold of under 5% free — there's no equivalent staged early warning the way there is for bytes. The CNCF post describes this as "kubelet watches inodes, just not until it's an emergency" (see the CNCF blog).

K3s inherits this same mechanism, so any K3s cluster running workloads that generate large numbers of small files — node_modules being the classic example — can fall into the same trap.

3. Diagnosing with df -i, and Understanding Why Hard Eviction Is So Dangerous

For the actual investigation once pods start getting evicted, a diagnostic guide from OneUptime, a Kubernetes observability provider, lays out concrete steps (see OneUptime: Diagnosing DiskPressure/Inode Evictions). The key points:

# Inspect the evicted pod
kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

# Check the node's DiskPressure condition
kubectl get node <node-name> -o json | jq '.status.conditions[] | select(.type == "DiskPressure")'

# ★ This is the critical step: check both bytes and inodes
df -h / /var/lib/kubelet /var/log
df -i / /var/lib/kubelet /var/log

If df -h shows plenty of free space but df -i shows inode usage near 100%, you've confirmed inode exhaustion as the cause.

Why hard eviction is genuinely scary

As the official Kubernetes documentation states, once a hard eviction threshold is crossed, kubelet terminates pods with a zero-second grace period. And unlike a normal API-driven eviction (such as kubectl drain), this node-pressure eviction does not respect PodDisruptionBudgets or a container's terminationGracePeriodSeconds at all (see Kubernetes docs: Node-pressure Eviction). In other words, any PDB you configured to protect availability becomes meaningless in the face of node resource pressure. That's exactly why the damage is often already widespread by the time anyone notices.

On top of that, eviction can also be triggered at the pod level if a pod exceeds its own ephemeral-storage limit or an emptyDir's sizeLimit (see Kubernetes docs: Local ephemeral storage). It's worth understanding that there are two separate paths here: node-wide pressure, and per-pod limit overruns.

4. The K3s/containerd-Specific Trap

K3s uses containerd as its container runtime by default. containerd unpacks image layers under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/, with each numbered directory corresponding to one image layer. As the CNCF post points out, it's not just node_modules — any image shipping a dependency tree made of huge numbers of small files, whether a Python virtualenv, a PHP vendor directory, or a pile of Ruby gems, can trigger the same problem (see the CNCF blog).

A K3s GitHub issue also documents a related case: on cgroup v2 systems, containerd-shim created an excessive number of inotify instances, triggering "too many open files" errors (see k3s-io/k3s Issue #10020). That's a distinct issue from inode exhaustion itself, but it shares the same root cause — file-related node resources quietly running out where nobody's watching. Disk space, inodes, and file descriptors are all the kind of thing that default monitoring dashboards tend to miss.

If you're running K3s in a small on-prem or edge environment, the stakes are higher: fewer nodes mean less redundancy, so a single node running out of inodes can take down the whole cluster's availability. For workloads with data residency requirements that rule out sending data off-site, an option like Kubo On-Premise can be deployed in an air-gapped environment with node monitoring built in.

5. Prevention: Multi-Stage Builds, Log Rotation, and Monitoring

Preventing eviction from inode exhaustion requires addressing both image design and monitoring.

Make the image lighter

Using multi-stage builds, as described in the official Docker documentation, you can separate a build-only stage from a runtime stage, keeping development-time dependencies like node_modules out of the production image entirely (see Docker docs: Multi-stage builds).

# Build stage
FROM node:22 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Runtime stage (no node_modules baggage)
FROM node:22-slim
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package*.json ./
RUN npm ci --production
CMD ["node", "dist/index.js"]

Put caps on logs and emptyDir

• Container log rotation: configure kubelet's containerLogMaxSize and containerLogMaxFiles
• Always set a sizeLimit on any pod using emptyDir, to prevent unbounded small-file creation

Monitoring and cleanup

• Alert on inode-related metrics such as node_filesystem_files_free separately from byte-based disk usage (80% is a reasonable warning threshold)
• In an emergency, crictl rmi --prune can bulk-remove unused images; one documented case cut disk usage from 71% down to 20% (see virtualizationhowto.com: Freeing Up Disk Space in a Kubernetes Cluster)

These measures aren't a one-off fix — the real design is two-pronged: "don't bring small files into the image at build time" and "make sure caps and alerts let you notice during operation." Managed K3s platforms like Kubo surface this kind of node resource monitoring directly in Captain UI, with alerts covering inode usage built in from the start. If you'd rather not design and maintain your own Prometheus alert rules for this, it's worth a look.

Summary

If you run into "pods keep getting evicted even though disk space looks fine" on K3s or Kubernetes, the first thing to suspect is inode exhaustion — something df -h simply can't show you. kubelet monitors six signals across nodefs, imagefs, and containerfs, but inodes lack the staged early warning that bytes get, and once a hard eviction threshold is crossed, pods get terminated regardless of PodDisruptionBudgets.

Because K3s's containerd overlayfs snapshots are prone to bloating from images full of small files, it's worth combining multi-stage builds to strip out dependencies, log rotation, sizeLimit on emptyDir, and monitoring of inode metrics.

If building and maintaining this kind of node resource visibility yourself feels like too much overhead, a managed option like Kubo Cloud is worth considering — it covers inode usage alongside disk capacity without you having to design your own Prometheus setup. Built on K3s, it also tends to be more cost-efficient than EKS/AKS/GKE while letting you keep running genuine Kubernetes, which is worth factoring into the decision. If that's of interest, feel free to reach out via Contact Us.