[{"data":1,"prerenderedAt":899},["ShallowReactive",2],{"blog-en-k3s-kubelet-eviction-inode-exhaustion-diskpressure":3,"blog-related-en-k3s-kubelet-eviction-inode-exhaustion-diskpressure":842,"blog-en-k3s-kubelet-eviction-inode-exhaustion-diskpressure-alt":408},{"id":4,"title":5,"author":6,"body":7,"date":825,"description":826,"extension":827,"image":828,"locale":829,"meta":830,"navigation":408,"path":831,"seo":832,"stem":833,"tags":834,"__hash__":841},"blog\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure.md","K3s Pods Kept Getting Evicted. Disk Space Was Fine — Inodes Were Not","Kubo Team",{"type":8,"value":9,"toc":805},"minimark",[10,18,38,54,62,68,83,106,111,126,130,136,149,247,264,268,278,284,292,298,307,499,508,512,534,550,554,560,573,586,595,599,605,608,612,623,708,712,733,737,759,766,770,779,788,801],[11,12,13],"p",{},[14,15],"img",{"alt":16,"src":17},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Fsection01.webp",[11,19,20,21,25,26,29,30,33,34,37],{},"One morning, pods across several nodes in a K3s cluster suddenly started getting evicted, one after another. Running ",[22,23,24],"code",{},"kubectl get pod -A"," showed a long list of pods marked ",[22,27,28],{},"Evicted",". The first suspect is always disk space. But ",[22,31,32],{},"df -h"," showed plenty of free space on every node. Yet the nodes stayed in a ",[22,35,36],{},"DiskPressure"," state, refusing to schedule new pods.",[11,39,40,41,45,46,53],{},"This article explains a frequently overlooked relationship in K3s\u002FKubernetes operations: the one between ",[42,43,44],"strong",{},"kubelet's eviction manager"," and inode exhaustion, walking through a realistic troubleshooting scenario. If you only watch disk capacity in bytes, you'll never spot the real reason pods keep getting evicted on K3s. This kind of \"invisible resource exhaustion\" is exactly the sort of thing that turns on-call response into tribal knowledge. If you're evaluating a managed K3s environment like ",[47,48,52],"a",{"href":49,"rel":50},"https:\u002F\u002Fkubo.hexabase.io\u002F",[51],"nofollow","Kubo",", the monitoring points covered here should give you a concrete checklist to judge it by.",[55,56,58,59,61],"h2",{"id":57},"_1-df-h-wont-show-the-culprit-inode-exhaustion-the-limit-on-file-creation-capacity","1. ",[22,60,32],{}," Won't Show the Culprit: Inode Exhaustion, the Limit on \"File-Creation Capacity\"",[11,63,64],{},[14,65],{"alt":66,"src":67},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Fsection02.webp",[11,69,70,71,74,75,78,79,82],{},"Linux filesystems maintain a separate accounting structure from disk capacity (bytes): the ",[42,72,73],{},"inode",". An inode is a data structure that records a file's or directory's metadata — owner, permissions, the location of its data blocks — and one inode is consumed every time a file is created. Filesystems like ext4 allocate a fixed number of inodes at format time, based on a ratio (the ",[22,76,77],{},"inode_ratio",", 16,384 bytes per inode by default), and that number ",[42,80,81],{},"cannot be changed later",".",[11,84,85,86,89,90,93,94,99,100,102,103,82],{},"In other words, a workload that stores a few large files has plenty of headroom, but a workload that ",[42,87,88],{},"creates huge numbers of small files"," will run out of inodes long before it runs out of bytes. In research published in October 2026, the CNCF — the foundation behind cloud native technologies — documented a real cluster where disk usage sat at 83% and inode usage at 67%, caused by a Node.js ",[22,91,92],{},"node_modules"," directory (21,553 files per snapshot) duplicated across multiple container image layers (see the ",[47,95,98],{"href":96,"rel":97},"https:\u002F\u002Fwww.cncf.io\u002Fblog\u002F2026\u002F10\u002F05\u002Fkubelet-watches-inodes-just-not-until-its-an-emergency\u002F",[51],"CNCF blog","). Per that post's calculation, with a standard ext4 ",[22,101,77],{},", inodes can run out at ",[42,104,105],{},"as little as 25% disk usage",[107,108,110],"h3",{"id":109},"why-its-so-easy-to-miss","Why it's so easy to miss",[112,113,114,120,123],"ul",{},[115,116,117,119],"li",{},[22,118,32],{}," only shows free capacity in bytes",[115,121,122],{},"Most monitoring dashboards aggregate disk usage by percentage of bytes",[115,124,125],{},"Processes that generate huge numbers of small files — logs, caches, temp files — rarely get a second thought",[55,127,129],{"id":128},"_2-the-six-signals-kubelet-watches-and-the-early-warning-that-doesnt-exist","2. The Six Signals kubelet Watches, and the Early Warning That Doesn't Exist",[11,131,132],{},[14,133],{"alt":134,"src":135},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Fsection03.webp",[11,137,138,139,142,143,148],{},"Per the official Kubernetes documentation, kubelet monitors multiple ",[42,140,141],{},"eviction signals"," to detect node resource pressure (see ",[47,144,147],{"href":145,"rel":146},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Fnode-pressure-eviction",[51],"Kubernetes docs: Node-pressure Eviction","). For disk and inodes alone, there are six distinct signals.",[150,151,152,168],"table",{},[153,154,155],"thead",{},[156,157,158,162,165],"tr",{},[159,160,161],"th",{},"Filesystem",[159,163,164],{},"Target",[159,166,167],{},"Signal",[169,170,171,185,197,210,222,235],"tbody",{},[156,172,173,177,180],{},[174,175,176],"td",{},"nodefs",[174,178,179],{},"Node's primary filesystem capacity",[174,181,182],{},[22,183,184],{},"nodefs.available",[156,186,187,189,192],{},[174,188,176],{},[174,190,191],{},"Node's primary filesystem inodes",[174,193,194],{},[22,195,196],{},"nodefs.inodesFree",[156,198,199,202,205],{},[174,200,201],{},"imagefs",[174,203,204],{},"Image store capacity",[174,206,207],{},[22,208,209],{},"imagefs.available",[156,211,212,214,217],{},[174,213,201],{},[174,215,216],{},"Image store inodes",[174,218,219],{},[22,220,221],{},"imagefs.inodesFree",[156,223,224,227,230],{},[174,225,226],{},"containerfs",[174,228,229],{},"Writable layer capacity",[174,231,232],{},[22,233,234],{},"containerfs.available",[156,236,237,239,242],{},[174,238,226],{},[174,240,241],{},"Writable layer inodes",[174,243,244],{},[22,245,246],{},"containerfs.inodesFree",[11,248,249,250,253,254,257,258,263],{},"According to a guide from the cloud native monitoring vendor Netdata, the default Linux hard eviction thresholds are ",[42,251,252],{},"nodefs: available capacity below 10%, inodes free below 5%",", and ",[42,255,256],{},"imagefs: available capacity below 15%, inodes free below 5%"," (see ",[47,259,262],{"href":260,"rel":261},"https:\u002F\u002Fwww.netdata.cloud\u002Fguides\u002Fkubernetes\u002Fkubernetes-node-disk-pressure\u002F",[51],"Netdata: Kubernetes node DiskPressure",").",[107,265,267],{"id":266},"theres-early-warning-for-bytes-but-none-for-inodes","There's early warning for bytes, but none for inodes",[11,269,270,271,274,275,263],{},"This is where a structural gap shows up. kubelet's image garbage collection kicks in once imagefs usage hits a byte-based high watermark of 85%, cleaning up unused images down to a low watermark of 80% — a graduated, byte-based early intervention. But for inodes, the ",[42,272,273],{},"only monitored point is the hard eviction threshold of under 5% free"," — there's no equivalent staged early warning the way there is for bytes. The CNCF post describes this as \"kubelet watches inodes, just not until it's an emergency\" (see the ",[47,276,98],{"href":96,"rel":277},[51],[11,279,280,281,283],{},"K3s inherits this same mechanism, so any K3s cluster running workloads that generate large numbers of small files — ",[22,282,92],{}," being the classic example — can fall into the same trap.",[55,285,287,288,291],{"id":286},"_3-diagnosing-with-df-i-and-understanding-why-hard-eviction-is-so-dangerous","3. Diagnosing with ",[22,289,290],{},"df -i",", and Understanding Why Hard Eviction Is So Dangerous",[11,293,294],{},[14,295],{"alt":296,"src":297},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Fsection04.webp",[11,299,300,301,306],{},"For the actual investigation once pods start getting evicted, a diagnostic guide from OneUptime, a Kubernetes observability provider, lays out concrete steps (see ",[47,302,305],{"href":303,"rel":304},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-08-27-diagnose-diskpressure-inode-evictions-ephemeral-storage\u002Fmarkdown",[51],"OneUptime: Diagnosing DiskPressure\u002FInode Evictions","). The key points:",[308,309,314],"pre",{"className":310,"code":311,"language":312,"meta":313,"style":313},"language-bash shiki shiki-themes tokyo-night","# Inspect the evicted pod\nkubectl describe pod \u003Cpod-name> -n \u003Cnamespace>\nkubectl get events -n \u003Cnamespace> --field-selector involvedObject.name=\u003Cpod-name>\n\n# Check the node's DiskPressure condition\nkubectl get node \u003Cnode-name> -o json | jq '.status.conditions[] | select(.type == \"DiskPressure\")'\n\n# ★ This is the critical step: check both bytes and inodes\ndf -h \u002F \u002Fvar\u002Flib\u002Fkubelet \u002Fvar\u002Flog\ndf -i \u002F \u002Fvar\u002Flib\u002Fkubelet \u002Fvar\u002Flog\n","bash","",[22,315,316,325,367,403,410,416,456,461,467,485],{"__ignoreMap":313},[317,318,321],"span",{"class":319,"line":320},"line",1,[317,322,324],{"class":323},"sbD-w","# Inspect the evicted pod\n",[317,326,328,332,336,339,343,346,350,353,357,359,362,364],{"class":319,"line":327},2,[317,329,331],{"class":330},"sE3pS","kubectl",[317,333,335],{"class":334},"sPY7s"," describe",[317,337,338],{"class":334}," pod",[317,340,342],{"class":341},"sAklC"," \u003C",[317,344,345],{"class":334},"pod-nam",[317,347,349],{"class":348},"sGX4V","e",[317,351,352],{"class":341},">",[317,354,356],{"class":355},"sT800"," -n",[317,358,342],{"class":341},[317,360,361],{"class":334},"namespac",[317,363,349],{"class":348},[317,365,366],{"class":341},">\n",[317,368,370,372,375,378,380,382,384,386,388,391,394,397,399,401],{"class":319,"line":369},3,[317,371,331],{"class":330},[317,373,374],{"class":334}," get",[317,376,377],{"class":334}," events",[317,379,356],{"class":355},[317,381,342],{"class":341},[317,383,361],{"class":334},[317,385,349],{"class":348},[317,387,352],{"class":341},[317,389,390],{"class":355}," --field-selector",[317,392,393],{"class":334}," involvedObject.name=",[317,395,396],{"class":341},"\u003C",[317,398,345],{"class":334},[317,400,349],{"class":348},[317,402,366],{"class":341},[317,404,406],{"class":319,"line":405},4,[317,407,409],{"emptyLinePlaceholder":408},true,"\n",[317,411,413],{"class":319,"line":412},5,[317,414,415],{"class":323},"# Check the node's DiskPressure condition\n",[317,417,419,421,423,426,428,431,433,435,438,441,444,447,450,453],{"class":319,"line":418},6,[317,420,331],{"class":330},[317,422,374],{"class":334},[317,424,425],{"class":334}," node",[317,427,342],{"class":341},[317,429,430],{"class":334},"node-nam",[317,432,349],{"class":348},[317,434,352],{"class":341},[317,436,437],{"class":355}," -o",[317,439,440],{"class":334}," json",[317,442,443],{"class":341}," |",[317,445,446],{"class":330}," jq",[317,448,449],{"class":341}," '",[317,451,452],{"class":334},".status.conditions[] | select(.type == \"DiskPressure\")",[317,454,455],{"class":341},"'\n",[317,457,459],{"class":319,"line":458},7,[317,460,409],{"emptyLinePlaceholder":408},[317,462,464],{"class":319,"line":463},8,[317,465,466],{"class":323},"# ★ This is the critical step: check both bytes and inodes\n",[317,468,470,473,476,479,482],{"class":319,"line":469},9,[317,471,472],{"class":330},"df",[317,474,475],{"class":355}," -h",[317,477,478],{"class":334}," \u002F",[317,480,481],{"class":334}," \u002Fvar\u002Flib\u002Fkubelet",[317,483,484],{"class":334}," \u002Fvar\u002Flog\n",[317,486,488,490,493,495,497],{"class":319,"line":487},10,[317,489,472],{"class":330},[317,491,492],{"class":355}," -i",[317,494,478],{"class":334},[317,496,481],{"class":334},[317,498,484],{"class":334},[11,500,501,502,504,505,507],{},"If ",[22,503,32],{}," shows plenty of free space but ",[22,506,290],{}," shows inode usage near 100%, you've confirmed inode exhaustion as the cause.",[107,509,511],{"id":510},"why-hard-eviction-is-genuinely-scary","Why hard eviction is genuinely scary",[11,513,514,515,518,519,522,523,257,530,533],{},"As the official Kubernetes documentation states, once a hard eviction threshold is crossed, kubelet terminates pods with a ",[42,516,517],{},"zero-second grace period",". And unlike a normal API-driven eviction (such as ",[22,520,521],{},"kubectl drain","), this node-pressure eviction ",[42,524,525,526,529],{},"does not respect PodDisruptionBudgets or a container's ",[22,527,528],{},"terminationGracePeriodSeconds"," at all",[47,531,147],{"href":145,"rel":532},[51],"). In other words, any PDB you configured to protect availability becomes meaningless in the face of node resource pressure. That's exactly why the damage is often already widespread by the time anyone notices.",[11,535,536,537,540,541,257,544,549],{},"On top of that, eviction can also be triggered at the pod level if a pod exceeds its own ephemeral-storage limit or an ",[22,538,539],{},"emptyDir","'s ",[22,542,543],{},"sizeLimit",[47,545,548],{"href":546,"rel":547},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fephemeral-storage\u002F",[51],"Kubernetes docs: Local ephemeral storage","). It's worth understanding that there are two separate paths here: node-wide pressure, and per-pod limit overruns.",[55,551,553],{"id":552},"_4-the-k3scontainerd-specific-trap","4. The K3s\u002Fcontainerd-Specific Trap",[11,555,556],{},[14,557],{"alt":558,"src":559},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Fsection05.webp",[11,561,562,563,566,567,569,570,263],{},"K3s uses containerd as its container runtime by default. containerd unpacks image layers under ",[22,564,565],{},"\u002Fvar\u002Flib\u002Fcontainerd\u002Fio.containerd.snapshotter.v1.overlayfs\u002Fsnapshots\u002F",", with each numbered directory corresponding to one image layer. As the CNCF post points out, it's not just ",[22,568,92],{}," — any image shipping a dependency tree made of huge numbers of small files, whether a Python virtualenv, a PHP vendor directory, or a pile of Ruby gems, can trigger the same problem (see the ",[47,571,98],{"href":96,"rel":572},[51],[11,574,575,576,579,580,585],{},"A K3s GitHub issue also documents a related case: on cgroup v2 systems, ",[22,577,578],{},"containerd-shim"," created an excessive number of inotify instances, triggering \"too many open files\" errors (see ",[47,581,584],{"href":582,"rel":583},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F10020",[51],"k3s-io\u002Fk3s Issue #10020","). That's a distinct issue from inode exhaustion itself, but it shares the same root cause — file-related node resources quietly running out where nobody's watching. Disk space, inodes, and file descriptors are all the kind of thing that default monitoring dashboards tend to miss.",[11,587,588,589,594],{},"If you're running K3s in a small on-prem or edge environment, the stakes are higher: fewer nodes mean less redundancy, so a single node running out of inodes can take down the whole cluster's availability. For workloads with data residency requirements that rule out sending data off-site, an option like ",[47,590,593],{"href":591,"rel":592},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[51],"Kubo On-Premise"," can be deployed in an air-gapped environment with node monitoring built in.",[55,596,598],{"id":597},"_5-prevention-multi-stage-builds-log-rotation-and-monitoring","5. Prevention: Multi-Stage Builds, Log Rotation, and Monitoring",[11,600,601],{},[14,602],{"alt":603,"src":604},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Fsection06.webp",[11,606,607],{},"Preventing eviction from inode exhaustion requires addressing both image design and monitoring.",[107,609,611],{"id":610},"make-the-image-lighter","Make the image lighter",[11,613,614,615,617,618,263],{},"Using multi-stage builds, as described in the official Docker documentation, you can separate a build-only stage from a runtime stage, keeping development-time dependencies like ",[22,616,92],{}," out of the production image entirely (see ",[47,619,622],{"href":620,"rel":621},"https:\u002F\u002Fdocs.docker.com\u002Fbuild\u002Fbuilding\u002Fmulti-stage\u002F",[51],"Docker docs: Multi-stage builds",[308,624,628],{"className":625,"code":626,"language":627,"meta":313,"style":313},"language-dockerfile shiki shiki-themes tokyo-night","# Build stage\nFROM node:22 AS builder\nWORKDIR \u002Fapp\nCOPY package*.json .\u002F\nRUN npm ci\nCOPY . .\nRUN npm run build\n\n# Runtime stage (no node_modules baggage)\nFROM node:22-slim\nWORKDIR \u002Fapp\nCOPY --from=builder \u002Fapp\u002Fdist .\u002Fdist\nCOPY --from=builder \u002Fapp\u002Fpackage*.json .\u002F\nRUN npm ci --production\nCMD [\"node\", \"dist\u002Findex.js\"]\n","dockerfile",[22,629,630,635,640,645,650,655,660,665,669,674,679,684,690,696,702],{"__ignoreMap":313},[317,631,632],{"class":319,"line":320},[317,633,634],{},"# Build stage\n",[317,636,637],{"class":319,"line":327},[317,638,639],{},"FROM node:22 AS builder\n",[317,641,642],{"class":319,"line":369},[317,643,644],{},"WORKDIR \u002Fapp\n",[317,646,647],{"class":319,"line":405},[317,648,649],{},"COPY package*.json .\u002F\n",[317,651,652],{"class":319,"line":412},[317,653,654],{},"RUN npm ci\n",[317,656,657],{"class":319,"line":418},[317,658,659],{},"COPY . .\n",[317,661,662],{"class":319,"line":458},[317,663,664],{},"RUN npm run build\n",[317,666,667],{"class":319,"line":463},[317,668,409],{"emptyLinePlaceholder":408},[317,670,671],{"class":319,"line":469},[317,672,673],{},"# Runtime stage (no node_modules baggage)\n",[317,675,676],{"class":319,"line":487},[317,677,678],{},"FROM node:22-slim\n",[317,680,682],{"class":319,"line":681},11,[317,683,644],{},[317,685,687],{"class":319,"line":686},12,[317,688,689],{},"COPY --from=builder \u002Fapp\u002Fdist .\u002Fdist\n",[317,691,693],{"class":319,"line":692},13,[317,694,695],{},"COPY --from=builder \u002Fapp\u002Fpackage*.json .\u002F\n",[317,697,699],{"class":319,"line":698},14,[317,700,701],{},"RUN npm ci --production\n",[317,703,705],{"class":319,"line":704},15,[317,706,707],{},"CMD [\"node\", \"dist\u002Findex.js\"]\n",[107,709,711],{"id":710},"put-caps-on-logs-and-emptydir","Put caps on logs and emptyDir",[112,713,714,724],{},[115,715,716,717,720,721],{},"Container log rotation: configure kubelet's ",[22,718,719],{},"containerLogMaxSize"," and ",[22,722,723],{},"containerLogMaxFiles",[115,725,726,727,729,730,732],{},"Always set a ",[22,728,543],{}," on any pod using ",[22,731,539],{},", to prevent unbounded small-file creation",[107,734,736],{"id":735},"monitoring-and-cleanup","Monitoring and cleanup",[112,738,739,746],{},[115,740,741,742,745],{},"Alert on inode-related metrics such as ",[22,743,744],{},"node_filesystem_files_free"," separately from byte-based disk usage (80% is a reasonable warning threshold)",[115,747,748,749,752,753,758],{},"In an emergency, ",[22,750,751],{},"crictl rmi --prune"," can bulk-remove unused images; one documented case cut disk usage from 71% down to 20% (see ",[47,754,757],{"href":755,"rel":756},"https:\u002F\u002Fwww.virtualizationhowto.com\u002F2025\u002F04\u002Ffree-up-disk-space-in-your-kubernetes-cluster\u002F",[51],"virtualizationhowto.com: Freeing Up Disk Space in a Kubernetes Cluster",")",[11,760,761,762,765],{},"These measures aren't a one-off fix — the real design is two-pronged: \"don't bring small files into the image at build time\" and \"make sure caps and alerts let you notice during operation.\" Managed K3s platforms like ",[47,763,52],{"href":49,"rel":764},[51]," surface this kind of node resource monitoring directly in Captain UI, with alerts covering inode usage built in from the start. If you'd rather not design and maintain your own Prometheus alert rules for this, it's worth a look.",[55,767,769],{"id":768},"summary","Summary",[11,771,772,773,775,776,82],{},"If you run into \"pods keep getting evicted even though disk space looks fine\" on K3s or Kubernetes, the first thing to suspect is inode exhaustion — something ",[22,774,32],{}," simply can't show you. kubelet monitors six signals across nodefs, imagefs, and containerfs, but inodes lack the staged early warning that bytes get, and once a hard eviction threshold is crossed, pods get terminated ",[42,777,778],{},"regardless of PodDisruptionBudgets",[11,780,781,782,784,785,787],{},"Because K3s's containerd overlayfs snapshots are prone to bloating from images full of small files, it's worth combining multi-stage builds to strip out dependencies, log rotation, ",[22,783,543],{}," on ",[22,786,539],{},", and monitoring of inode metrics.",[11,789,790,791,795,796,82],{},"If building and maintaining this kind of node resource visibility yourself feels like too much overhead, a managed option like ",[47,792,794],{"href":49,"rel":793},[51],"Kubo Cloud"," is worth considering — it covers inode usage alongside disk capacity without you having to design your own Prometheus setup. Built on K3s, it also tends to be more cost-efficient than EKS\u002FAKS\u002FGKE while letting you keep running genuine Kubernetes, which is worth factoring into the decision. If that's of interest, feel free to reach out via ",[47,797,800],{"href":798,"rel":799},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[51],"Contact Us",[802,803,804],"style",{},"html pre.shiki code .sbD-w, html code.shiki .sbD-w{--shiki-default:#51597D;--shiki-default-font-style:italic}html pre.shiki code .sE3pS, html code.shiki .sE3pS{--shiki-default:#C0CAF5}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sGX4V, html code.shiki .sGX4V{--shiki-default:#A9B1D6}html pre.shiki code .sT800, html code.shiki .sT800{--shiki-default:#E0AF68}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":313,"searchDepth":327,"depth":327,"links":806},[807,811,814,818,819,824],{"id":57,"depth":327,"text":808,"children":809},"1. df -h Won't Show the Culprit: Inode Exhaustion, the Limit on \"File-Creation Capacity\"",[810],{"id":109,"depth":369,"text":110},{"id":128,"depth":327,"text":129,"children":812},[813],{"id":266,"depth":369,"text":267},{"id":286,"depth":327,"text":815,"children":816},"3. Diagnosing with df -i, and Understanding Why Hard Eviction Is So Dangerous",[817],{"id":510,"depth":369,"text":511},{"id":552,"depth":327,"text":553},{"id":597,"depth":327,"text":598,"children":820},[821,822,823],{"id":610,"depth":369,"text":611},{"id":710,"depth":369,"text":711},{"id":735,"depth":369,"text":736},{"id":768,"depth":327,"text":769},"2026-11-08","K3s pods kept getting evicted while df -h looked perfectly healthy. The real cause wasn't disk space but inode exhaustion. We cover kubelet's eviction thresholds, how to diagnose with df -i, and how to prevent it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure\u002Ftitle.webp","en",{},"\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure",{"title":5,"description":826},"blog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure",[835,836,837,838,839,840],"k3s","kubernetes","kubelet","eviction","diskpressure","troubleshooting","0IaR6rb0qbsz-K2Jp64GXm4Vn2yUodbXjwwCz7nbmQs",[843,851,860,870,880,890],{"path":844,"title":845,"description":846,"date":847,"tags":848},"\u002Fblog\u002Fen\u002Fk3s-rootless-kubelet-user-namespace-k8s-137","The Day KubeletInUserNamespace Went Beta: Is K3s Rootless Mode Now Obsolete?","Kubernetes 1.37 promoted KubeletInUserNamespace (Rootless Mode) to Beta. K3s has had its own Rootless Mode for years, but the mechanism is completely different. Here's how the two approaches compare and how to choose between them.","2026-10-28",[835,836,849,850,837],"rootless","security",{"path":852,"title":853,"description":854,"date":855,"tags":856},"\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction","A Resource Optimizer Broke K3s: All Replicas Evicted at Once. PodDisruptionBudget as the Last Line of Defense","A real incident where enabling automated resource optimization on K3s caused all replicas to be evicted simultaneously, triggering a full outage. This post explains why pods get evicted en masse and how PodDisruptionBudget's minAvailable setting prevents it.","2026-09-18",[835,836,857,838,858,859],"poddisruptionbudget","vertical-scaling","resource-optimization",{"path":861,"title":862,"description":863,"date":864,"tags":865},"\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap","We Couldn't Reconstruct What an AI Agent Did to Our Kubernetes API. The Gap Was in Our Audit Log Settings","After running AI agents on Kubernetes, we realized that when something went wrong, the Audit Log alone couldn't tell us who did what, or why. Here's the Audit Policy blind spot and how we fixed agent traceability.","2026-11-09",[835,836,866,867,868,869],"ai-agent","audit-log","rbac","observability",{"path":871,"title":872,"description":873,"date":874,"tags":875},"\u002Fblog\u002Fen\u002Fk3s-cilium-kube-proxy-replacement-nodeport-unreachable","We Swapped kube-proxy for Cilium on K3s, and NodePort Stopped Working","We followed the official Cilium guide to enable kube-proxy replacement on K3s, and NodePort traffic suddenly stopped reaching our pods. The cause: K3s-specific defaults (Flannel, ServiceLB) that the generic docs never mention. Here's the fix and a pre-production checklist.","2026-11-07",[835,836,876,877,878,879],"cilium","ebpf","kube-proxy","networking",{"path":881,"title":882,"description":883,"date":884,"tags":885},"\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling","requests and limits were correct, but my K3s Pod got slow anyway. The culprit wasn't OOMKill — it was cgroup v2's memory.high throttling early","A K3s Pod with correctly configured requests\u002Flimits suddenly got slow, with no OOMKilled event anywhere. The real cause was cgroup v2's memory.high throttling. Here's how to diagnose and fix it.","2026-11-06",[835,836,886,887,888,889],"cgroup-v2","memory-qos","memory-high","resource-management",{"path":891,"title":892,"description":893,"date":894,"tags":895},"\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied","The Volume That Worked Fine in Docker Compose Threw Permission Denied the Moment It Hit K3s","Migrating from Docker Compose to K3s? A volume that worked fine locally suddenly throws Permission denied. The cause is local-path-provisioner's design. Here's how to work around it.","2026-11-05",[835,836,896,897,898],"docker-compose","local-path-provisioner","persistent-storage",1794305514050]