[{"data":1,"prerenderedAt":539},["ShallowReactive",2],{"blog-en-k3s-local-path-provisioner-capacity-node-affinity-pvc-pending":3,"blog-related-en-k3s-local-path-provisioner-capacity-node-affinity-pvc-pending":484,"blog-en-k3s-local-path-provisioner-capacity-node-affinity-pvc-pending-alt":471},{"id":4,"title":5,"author":6,"body":7,"date":465,"description":466,"extension":467,"image":468,"locale":469,"meta":470,"navigation":471,"path":472,"seo":473,"stem":474,"tags":475,"__hash__":483},"blog\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending.md","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","Kubo Team",{"type":8,"value":9,"toc":448},"minimark",[10,15,23,35,47,50,54,60,79,95,98,103,128,135,144,148,154,160,179,185,191,195,216,224,228,234,237,260,269,285,289,304,308,314,317,330,342,351,371,374,378,384,394,397,423,432,439],[11,12,14],"h2",{"id":13},"_1-one-day-a-k3s-pod-simply-stopped-coming-back-from-pending","1. One Day, a K3s Pod Simply Stopped Coming Back From Pending",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection01.webp",[16,24,25,26,30,31,34],{},"You created a new PVC on your K3s cluster, but the Pod sits in ",[27,28,29],"code",{},"Pending"," for tens of minutes and never reaches ",[27,32,33],{},"Running",". If you've run K3s in production for any length of time, you've probably hit this at least once.",[16,36,37,38,41,42,46],{},"Running ",[27,39,40],{},"kubectl describe pod"," shows nothing wrong with image pulls or scheduling itself. In practice, the culprit is often something quieter: ",[43,44,45],"strong",{},"the node's local disk capacity had been silently filling up."," In architectures that lean on node-local storage, like K3s typically does, this kind of invisible capacity trap is a recurring source of production incidents.",[16,48,49],{},"This article works through four commonly overlooked aspects of K3s local storage design, in order: a design constraint in K3s's default storage mechanism (local-path-provisioner), hypervisor-side thin provisioning on platforms like Proxmox, the Volume Node Affinity Conflict that appears when nodes change, and the difference between a snapshot and an actual backup.",[11,51,53],{"id":52},"_2-local-path-provisioner-doesnt-actually-look-at-pvc-capacity","2. local-path-provisioner Doesn't Actually Look at PVC \"Capacity\"",[16,55,56],{},[19,57],{"alt":58,"src":59},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection02.webp",[16,61,62,63,70,71,78],{},"K3s ships with Rancher's ",[64,65,69],"a",{"href":66,"rel":67},"https:\u002F\u002Fgithub.com\u002Francher\u002Flocal-path-provisioner",[68],"nofollow","Local Path Provisioner"," out of the box. By default, persistent volumes are created on the node's own filesystem, under ",[64,72,75],{"href":73,"rel":74},"https:\u002F\u002Fdocs.k3s.io\u002Fadd-ons\u002Fstorage",[68],[27,76,77],{},"\u002Fvar\u002Flib\u002Francher\u002Fk3s\u002Fstorage",". It's a convenient way to get PersistentVolumes working with almost no setup — but it comes with a design constraint that's easy to miss.",[16,80,81,84,85,88,89,94],{},[43,82,83],{},"local-path-provisioner does not actually enforce the capacity you declare on a PVC."," This is acknowledged directly in discussion on Rancher's own GitHub repository. One issue thread notes that \"since local-path-provisioner doesn't enforce capacity limitations, there's no reason to throw errors\" when someone changes ",[27,86,87],{},"spec.capacity"," — ",[64,90,93],{"href":91,"rel":92},"https:\u002F\u002Fgithub.com\u002Francher\u002Flocal-path-provisioner\u002Fissues\u002F190",[68],"confirming that capacity limits are effectively ignored by design",".",[16,96,97],{},"In other words, writing \"10Gi\" on a PVC is just a declaration at the Kubernetes API level. If a container keeps writing more data than that, local-path-provisioner won't stop it. The data keeps consuming the node's physical disk, eventually squeezing out space that other Pods depend on.",[99,100,102],"h3",{"id":101},"what-actually-happens","What Actually Happens",[104,105,106,114,121],"ul",{},[107,108,109,110,113],"li",{},"An application can keep writing logs or cache data past its ",[27,111,112],{},"requests.storage"," value without ever hitting an error",[107,115,116,117,120],{},"As node disk usage approaches 100%, the kubelet raises ",[27,118,119],{},"DiskPressure"," on the node, which starts evicting existing Pods and refusing to schedule new ones",[107,122,123,124,127],{},"The overrun never shows up in the PVC's status. ",[43,125,126],{},"What you actually need to watch is per-node disk usage"," — the PVC's declared value alone won't warn you",[16,129,130,131,134],{},"Volume Expansion (",[27,132,133],{},"AllowVolumeExpansion",") support has also long been deferred in local-path-provisioner, so in practice you have to assume capacity management is something you monitor and provision for at the node level.",[16,136,137,138,143],{},"Closing this gap between declared and actual usage with a homegrown monitoring dashboard is tedious work. With ",[64,139,142],{"href":140,"rel":141},"https:\u002F\u002Fkubo.hexabase.io\u002F",[68],"Kubo","'s Captain UI, per-node resource status is visible directly from the cluster management screen, which makes it much easier to spot these capacity warning signs early.",[11,145,147],{"id":146},"_3-the-overlooked-second-factor-hypervisor-side-thin-provisioning","3. The Overlooked Second Factor: Hypervisor-Side Thin Provisioning",[16,149,150],{},[19,151],{"alt":152,"src":153},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection03.webp",[16,155,156,157],{},"If your K3s nodes are VMs running on a hypervisor like Proxmox VE, things get another layer more complicated. On top of K3s ignoring capacity, ",[43,158,159],{},"the hypervisor's own storage can have a nearly identical \"invisible capacity\" problem.",[16,161,162,163,168,169,174,175,178],{},"Proxmox VE's default storage backends, ZFS and LVM-Thin, both support thin provisioning. According to ",[64,164,167],{"href":165,"rel":166},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FStorage:_LVM_Thin",[68],"Proxmox's official wiki",", with this approach \"volumes can be much larger than physically available space.\" Similarly, ",[64,170,173],{"href":171,"rel":172},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FStorage:_ZFS",[68],"the official ZFS storage documentation"," explains that enabling the ",[27,176,177],{},"sparse"," property creates sparse volumes \"whose reservation is not equal to the volume size.\"",[16,180,181,182],{},"In other words, even if you assign a 100GB virtual disk to each of several K3s node VMs, the pool only actually consumes physical capacity equal to the data that has really been written. That's an efficient design — but it also means ",[43,183,184],{},"multiple VMs sharing the same storage pool can easily create an \"overcommit\" state, where the sum of declared virtual capacity exceeds the pool's physical capacity.",[16,186,187,188],{},"When the thin pool's physical capacity actually runs out, writes start failing abruptly, even though the VM (the K3s node) still appears to have free disk space. With K3s's local-path-provisioner not watching capacity, and the hypervisor's pool exhaustion being just as hard to notice, it's a natural consequence that ",[43,189,190],{},"the failure develops while remaining invisible on two separate layers at once.",[99,192,194],{"id":193},"practical-mitigations","Practical Mitigations",[104,196,197,204,213],{},[107,198,199,200,203],{},"Monitor hypervisor storage pool usage ",[43,201,202],{},"at the pool level",", not per-VM",[107,205,206,207,212],{},"As ",[64,208,211],{"href":209,"rel":210},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FStorage",[68],"Proxmox's storage overview page"," notes, ZFS and LVM-Thin are node-local, non-shared storage; if you need live migration or redundancy across multiple nodes, moving to distributed storage like Ceph\u002FRBD is an option worth considering",[107,214,215],{},"Design capacity monitoring for K3s and the hypervisor as two independent concerns",[16,217,218,219,223],{},"Tracking when to rebuild a node or relocate a VM, and which PV is pinned to which node, by hand is a recipe for tribal knowledge. ",[64,220,222],{"href":140,"rel":221},[68],"Kubo Cloud"," ships with a Rancher-based management layer as standard, which makes node-to-volume relationships visible and helps prevent this class of incident from recurring.",[11,225,227],{"id":226},"_4-crossing-nodes-breaks-everything-the-real-nature-of-volume-node-affinity-conflict","4. Crossing Nodes Breaks Everything: The Real Nature of Volume Node Affinity Conflict",[16,229,230],{},[19,231],{"alt":232,"src":233},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection04.webp",[16,235,236],{},"There's another classic dead end with node-local storage: the \"Volume Node Affinity Conflict.\"",[16,238,239,240,243,244,249,250,253,254,259],{},"The ",[27,241,242],{},"local"," volume type defined in the ",[64,245,248],{"href":246,"rel":247},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fvolumes\u002F",[68],"official Kubernetes documentation"," is, by nature, storage that \"exists only on a specific node,\" and a PersistentVolume using it ",[43,251,252],{},"requires a node affinity (nodeAffinity) specification."," ",[64,255,258],{"href":256,"rel":257},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fpersistent-volumes\u002F",[68],"The PersistentVolume documentation"," likewise lists Node Affinity as one of the key configuration fields for a volume.",[16,261,262,263,266,267,94],{},"The flip side of this is that ",[43,264,265],{},"once a PV is tied to a given node, a Pod using it can only be scheduled on that same node."," If you relocate a VM to a different physical host on Proxmox, or rebuild a K3s node from scratch, disk paths or UUIDs can change — and the mismatch between the PV's declared node affinity and the Pod's actual scheduling target leaves the Pod stuck in ",[27,268,29],{},[16,270,271,272,277,278,281,282],{},"This constraint has been a long-standing operational pain point, but there's an interesting development worth noting. According to the ",[64,273,276],{"href":274,"rel":275},"https:\u002F\u002Fwww.kubernetes.io\u002Fblog\u002F2026\u002F01\u002F08\u002Fkubernetes-v1-35-mutable-pv-nodeaffinity\u002F",[68],"Kubernetes blog"," published in January 2026, ",[43,279,280],{},"Kubernetes v1.35 added the ability to make a PersistentVolume's node affinity mutable, as an alpha feature."," The post states plainly that \"this field was previously immutable, and it is now mutable in Kubernetes v1.35 (alpha),\" explaining that this lets administrators update node affinity as underlying storage changes (for example, migration to regional disks). Read the other way, ",[43,283,284],{},"this change is itself evidence of how real a problem the immutability of PV node affinity has been in practice.",[99,286,288],{"id":287},"practical-workarounds","Practical Workarounds",[104,290,291,294,301],{},[107,292,293],{},"If you use node-local PVs, build PV recreation into your operational workflow for VM relocation and node replacement",[107,295,296,297,300],{},"Use ",[27,298,299],{},"volumeBindingMode: WaitForFirstConsumer"," on your StorageClass so the PV only binds after the Pod's scheduling target is decided, reducing unnecessary mismatches",[107,302,303],{},"For workloads that genuinely need portability across nodes, consider not using node-local storage at all",[11,305,307],{"id":306},"_5-a-zfs-snapshot-is-not-a-backup","5. A ZFS Snapshot Is Not a \"Backup\"",[16,309,310],{},[19,311],{"alt":312,"src":313},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection05.webp",[16,315,316],{},"Even once you've worked through capacity and scheduling issues, there's one more misconception worth untangling: the belief that \"as long as I have a snapshot, my backup story is complete.\"",[16,318,319,320,325,326,329],{},"A ZFS snapshot, according to the ",[64,321,324],{"href":322,"rel":323},"https:\u002F\u002Fopenzfs.github.io\u002Fopenzfs-docs\u002Fman\u002Fmaster\u002F8\u002Fzfs-snapshot.8.html",[68],"official OpenZFS documentation",", is defined as \"a consistent image of a dataset at a specific point in time... that includes all modifications to the dataset made by system calls that have successfully completed before that point in time.\" That's a powerful capability, but it remains ",[43,327,328],{},"a point-in-time copy at the storage layer"," — it does not protect the state of your Kubernetes cluster itself (the resource definitions stored in etcd).",[16,331,332,333,338,339],{},"Kubernetes cluster state lives in ",[64,334,337],{"href":335,"rel":336},"https:\u002F\u002Fetcd.io\u002Fdocs\u002Fv3.5\u002Fop-guide\u002Frecovery\u002F",[68],"etcd",", and the official documentation explains that \"etcd v3 provides snapshot and restore facilities to recreate the cluster without v3 key data loss\" for disaster recovery. In other words, an etcd snapshot ",[43,340,341],{},"protects your cluster's configuration (Deployments, Services, Secrets, and so on) — it does not protect the actual data sitting in your PVCs\u002FPVs.",[16,343,344,345,350],{},"Meanwhile, the ",[64,346,349],{"href":347,"rel":348},"https:\u002F\u002Fvelero.io\u002Fdocs\u002Fmain\u002Fhow-velero-works\u002F",[68],"official Velero documentation"," explains that Velero archives Kubernetes objects to object storage while also, when configured, \"calling the cloud provider API to make disk snapshots of persistent volumes.\" Velero is therefore a broader protection layer that can cover both API resources and PV data.",[104,352,353,359,365],{},[107,354,355,358],{},[43,356,357],{},"Hypervisor snapshot"," (e.g., a Proxmox snapshot): a point-in-time copy of the VM disk. Useful for recovering a single K3s node, but it doesn't guarantee consistency across the whole cluster",[107,360,361,364],{},[43,362,363],{},"etcd snapshot",": protects cluster configuration. Does not cover the actual data in PVCs\u002FPVs",[107,366,367,370],{},[43,368,369],{},"Velero",": can cover both API resources and PV data, but requires its own separate setup and operation",[16,372,373],{},"These three layers play different roles. Relying on just one of them is exactly how you end up discovering, mid-incident, that the thing you actually needed to restore isn't what you backed up.",[11,375,377],{"id":376},"_6-summary-what-to-check-before-you-trust-node-local-storage","6. Summary: What to Check Before You Trust Node-Local Storage",[16,379,380],{},[19,381],{"alt":382,"src":383},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection06.webp",[16,385,386,387,389,390,393],{},"A K3s PVC stuck in ",[27,388,29],{}," rarely comes down to a simple misconfiguration. More often, the real cause is the fact that ",[43,391,392],{},"local-path-provisioner was never designed to watch capacity"," in the first place. Layer in hypervisor thin provisioning, node affinity constraints, and misunderstandings about backups, and the root cause becomes even harder to pin down.",[16,395,396],{},"Here's a quick operational checklist covering the four points this article walked through:",[104,398,399,405,411,417],{},[107,400,401,404],{},[43,402,403],{},"Capacity monitoring",": Are you watching per-node disk usage, rather than trusting the PVC's declared capacity?",[107,406,407,410],{},[43,408,409],{},"Thin provisioning monitoring",": Are you checking the hypervisor's storage pool at the pool level, not just per-VM usage?",[107,412,413,416],{},[43,414,415],{},"Node affinity operations",": Does your operational workflow assume that rebuilding a node or relocating a VM can strand a PV?",[107,418,419,422],{},[43,420,421],{},"Three-layer backups",": Do you understand the different roles of hypervisor snapshots, etcd snapshots, and Velero (including PV data), and are you combining the layers you actually need?",[16,424,425,426,431],{},"Designing and monitoring all of this yourself on a self-managed Proxmox + K3s environment is real, ongoing work. If you'd rather hand off storage design as part of your cluster operations, ",[64,427,430],{"href":428,"rel":429},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[68],"Kubo On-Premise"," is worth a look — it lets you build on your own infrastructure, Proxmox included, while getting fully managed Kubernetes operations that account for exactly this kind of storage design and monitoring.",[16,433,434,435,438],{},"And if you'd rather start by trying a managed K8s on the public cloud, ",[64,436,142],{"href":140,"rel":437},[68]," Cloud bakes in the same storage design know-how. It's built on lightweight K3s, and keeps monthly costs lower than EKS or AKS — a difference that matters even more once you factor in the operational design work this article has walked through.",[16,440,441,442,447],{},"Whether you keep fighting node-local storage traps yourself or hand the whole design over to a managed service is a real trade-off. If you're not sure which way to go, feel free to reach out via ",[64,443,446],{"href":444,"rel":445},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[68],"our contact page"," and talk it through.",{"title":449,"searchDepth":450,"depth":450,"links":451},"",2,[452,453,457,460,463,464],{"id":13,"depth":450,"text":14},{"id":52,"depth":450,"text":53,"children":454},[455],{"id":101,"depth":456,"text":102},3,{"id":146,"depth":450,"text":147,"children":458},[459],{"id":193,"depth":456,"text":194},{"id":226,"depth":450,"text":227,"children":461},[462],{"id":287,"depth":456,"text":288},{"id":306,"depth":450,"text":307},{"id":376,"depth":450,"text":377},"2026-09-12","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending",{"title":5,"description":466},"blog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending",[476,477,478,479,480,481,482],"k3s","kubernetes","storage","local-path-provisioner","persistent-volume","node-affinity","zfs","-xLekODJ-ssd31qnKRziLtWG8W9RmbkHfQfLPhn1uuE",[485,494,503,511,519,529],{"path":486,"title":487,"description":488,"date":489,"tags":490},"\u002Fblog\u002Fen\u002Fk3s-longhorn-openebs-mayastor-storage-selection","Is Longhorn Enough, or Do You Need OpenEBS Mayastor? Deciding K3s Storage by Performance Requirements","Still defaulting to Longhorn for K3s storage without a second thought? We compare Longhorn and OpenEBS Mayastor across hardware requirements, replica configuration, and performance characteristics, with a checklist for choosing without regret.","2026-11-15",[476,477,491,492,493,478],"longhorn","openebs","mayastor",{"path":495,"title":496,"description":497,"date":498,"tags":499},"\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance","Multi-Attach error for volume: What to Know About RBD Locks Before Node Maintenance on Proxmox K3s + Ceph CSI","Draining a node on a Proxmox-hosted K3s cluster left a Ceph CSI (RBD) PV stuck in Multi-Attach error. The cause: a stale exclusive lock, a lingering watcher, and a fencing race condition. Recovery commands and a permanent NetworkFence fix inside.","2026-11-14",[476,477,500,501,502,478],"proxmox","ceph-csi","rbd",{"path":504,"title":505,"description":506,"date":507,"tags":508},"\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied","The Volume That Worked Fine in Docker Compose Threw Permission Denied the Moment It Hit K3s","Migrating from Docker Compose to K3s? A volume that worked fine locally suddenly throws Permission denied. The cause is local-path-provisioner's design. Here's how to work around it.","2026-11-05",[476,477,509,479,510],"docker-compose","persistent-storage",{"path":512,"title":513,"description":514,"date":515,"tags":516},"\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum","A K3s Cluster Built on 2 Proxmox Hosts Went Down Entirely After a Brief Network Blip. The Culprit Was Corosync's Self-Fencing","A K3s cluster running on 2 Proxmox hosts went down completely after a few seconds of network interruption: both the host and the etcd quorum were taken out together. Here's why Corosync's self-fencing was the real cause, and how node placement design prevents it.","2026-11-11",[476,477,500,337,517,518],"high-availability","quorum",{"path":520,"title":521,"description":522,"date":523,"tags":524},"\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap","We Couldn't Reconstruct What an AI Agent Did to Our Kubernetes API. The Gap Was in Our Audit Log Settings","After running AI agents on Kubernetes, we realized that when something went wrong, the Audit Log alone couldn't tell us who did what, or why. Here's the Audit Policy blind spot and how we fixed agent traceability.","2026-11-09",[476,477,525,526,527,528],"ai-agent","audit-log","rbac","observability",{"path":530,"title":531,"description":532,"date":533,"tags":534},"\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure","K3s Pods Kept Getting Evicted. Disk Space Was Fine — Inodes Were Not","K3s pods kept getting evicted while df -h looked perfectly healthy. The real cause wasn't disk space but inode exhaustion. We cover kubelet's eviction thresholds, how to diagnose with df -i, and how to prevent it.","2026-11-08",[476,477,535,536,537,538],"kubelet","eviction","diskpressure","troubleshooting",1794902500908]