[{"data":1,"prerenderedAt":620},["ShallowReactive",2],{"blog-en-k3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance":3,"blog-related-en-k3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance":564,"blog-en-k3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance-alt":553},{"id":4,"title":5,"author":6,"body":7,"date":547,"description":548,"extension":549,"image":550,"locale":551,"meta":552,"navigation":553,"path":554,"seo":555,"stem":556,"tags":557,"__hash__":563},"blog\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance.md","Multi-Attach error for volume: What to Know About RBD Locks Before Node Maintenance on Proxmox K3s + Ceph CSI","Kubo Team",{"type":8,"value":9,"toc":539},"minimark",[10,18,23,30,37,47,71,80,84,90,103,110,126,130,136,142,214,223,332,341,348,389,392,396,402,415,441,462,471,475,481,494,506,510,513,522,535],[11,12,13,17],"p",{},[14,15,16],"code",{},"Multi-Attach error for volume",". If you run a self-hosted K3s cluster on Proxmox with Ceph CSI (RBD) as persistent storage, you've probably seen this message and felt your stomach drop. A routine, planned node maintenance, and suddenly a Pod won't come back from Pending. This article breaks down the cause, the manual recovery steps, and a permanent fix.",[19,20,22],"h2",{"id":21},"_1-right-after-draining-a-node-a-pod-got-stuck-in-containercreating","1. Right after draining a node, a Pod got stuck in \"ContainerCreating\"",[11,24,25],{},[26,27],"img",{"alt":28,"src":29},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance\u002Fsection01.webp",[11,31,32,33,36],{},"A kernel update and some Proxmox host maintenance called for draining one node in the K3s cluster — ",[14,34,35],{},"kubectl drain",", same as always. But right after the Pod got rescheduled onto another node, this line showed up in the event log:",[38,39,44],"pre",{"className":40,"code":42,"language":43},[41],"language-text","Warning  FailedAttachVolume  pod\u002Fapp-xxxx  Multi-Attach error for volume \"pvc-xxxx\" Volume is already exclusively attached to one node and can't be attached to another\n","text",[14,45,42],{"__ignoreMap":46},"",[11,48,49,50,57,58,61,62,66,67,70],{},"Per the ",[51,52,56],"a",{"href":53,"rel":54},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fpersistent-volumes\u002F",[55],"nofollow","official Kubernetes documentation",", ",[14,59,60],{},"ReadWriteOnce"," (RWO) means \"mountable read-write by a single node.\" Multiple Pods on the same node can share it fine, but ",[63,64,65],"strong",{},"it can't be attached to two different nodes at once",". ",[14,68,69],{},"ReadWriteMany"," (RWX, supported by CephFS and similar) allows concurrent mounts across nodes — so this issue is specific to block storage like RBD running in RWO mode.",[11,72,73,74,79],{},"The Pod on the drained node is long gone. So why can't the new node grab the volume? The answer lies in Ceph's own locking mechanism. That kind of fencing design is something every team running a self-hosted K3s cluster eventually has to confront — and it's one of the things worth weighing when evaluating a ",[51,75,78],{"href":76,"rel":77},"https:\u002F\u002Fkubo.hexabase.io\u002F",[55],"managed Kubernetes"," option instead.",[19,81,83],{"id":82},"_2-the-cause-a-lingering-rbd-exclusive-lock-and-watcher-plus-a-kubernetes-race-condition","2. The cause: a lingering RBD \"exclusive lock\" and watcher, plus a Kubernetes race condition",[11,85,86],{},[26,87],{"alt":88,"src":89},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance\u002Fsection02.webp",[11,91,92,93,98,99,102],{},"Ceph RBD images have the ",[51,94,97],{"href":95,"rel":96},"https:\u002F\u002Fdocs.ceph.com\u002Fen\u002Freef\u002Frbd\u002Frbd-exclusive-locks\u002F",[55],"exclusive-lock feature"," enabled by default. It guarantees that \"only a single client can write at any given time,\" and under normal process termination, the official Ceph documentation explains that ",[63,100,101],{},"the lock transitions cooperatively between clients",".",[11,104,105,106,109],{},"The trouble starts when a node doesn't terminate gracefully — a drain that times out, a kubelet that stops responding, a forced host reboot. In those cases, the old client disappears without releasing its lock. When a new client tries to acquire it, Ceph ",[63,107,108],{},"blocklists the old client and forcibly breaks the lock"," — but if Kubernetes reschedules the Pod onto another node before that blocklisting completes, the CSI driver tries to attach while the lock is still technically held, and you get a Multi-Attach error.",[11,111,112,113,116,117,122,123,125],{},"There's a second, nastier layer: the timing of ",[14,114,115],{},"VolumeAttachment"," object garbage collection. The ",[51,118,121],{"href":119,"rel":120},"https:\u002F\u002Frook.io\u002Fdocs\u002Frook\u002Fv1.9\u002Fceph-csi-troubleshooting.html",[55],"official Rook Ceph CSI troubleshooting guide"," notes that ",[14,124,115],{}," objects can linger even after communication with a failed node is lost. When the scheduler tries to re-attach on a new node before Ceph's actual fencing has completed, that asynchronous timing gap is the direct trigger for Multi-Attach errors.",[19,127,129],{"id":128},"_3-manual-recovery-from-deleting-the-volumeattachment-to-releasing-the-lock","3. Manual recovery: from deleting the VolumeAttachment to releasing the lock",[11,131,132],{},[26,133],{"alt":134,"src":135},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance\u002Fsection03.webp",[11,137,138,139,141],{},"Only if you're certain the failed node is completely down, the following manual recovery steps apply. First check for a lingering ",[14,140,115],{},", then force-delete the Pod if it's stuck Terminating on the old node.",[38,143,147],{"className":144,"code":145,"language":146,"meta":46,"style":46},"language-bash shiki shiki-themes tokyo-night","kubectl get volumeattachment | grep \u003Cpvc-name>\nkubectl delete pod \u003Cstuck-pod> --grace-period=0 --force\n","bash",[14,148,149,185],{"__ignoreMap":46},[150,151,154,158,162,165,169,172,175,178,182],"span",{"class":152,"line":153},"line",1,[150,155,157],{"class":156},"sE3pS","kubectl",[150,159,161],{"class":160},"sPY7s"," get",[150,163,164],{"class":160}," volumeattachment",[150,166,168],{"class":167},"sAklC"," |",[150,170,171],{"class":156}," grep",[150,173,174],{"class":167}," \u003C",[150,176,177],{"class":160},"pvc-nam",[150,179,181],{"class":180},"sGX4V","e",[150,183,184],{"class":167},">\n",[150,186,188,190,193,196,198,201,204,207,211],{"class":152,"line":187},2,[150,189,157],{"class":156},[150,191,192],{"class":160}," delete",[150,194,195],{"class":160}," pod",[150,197,174],{"class":167},[150,199,200],{"class":160},"stuck-po",[150,202,203],{"class":180},"d",[150,205,206],{"class":167},">",[150,208,210],{"class":209},"sT800"," --grace-period=0",[150,212,213],{"class":209}," --force\n",[11,215,216,217,222],{},"As ",[51,218,221],{"href":219,"rel":220},"https:\u002F\u002Faccess.redhat.com\u002Fnode\u002F7000129",[55],"Red Hat's support documentation"," also reports, an RWO ceph-rbd volume can remain unmountable by a new Pod even with no running Pod attached to it. Next, check Ceph's lock state directly and force-release it.",[38,224,226],{"className":144,"code":225,"language":146,"meta":46,"style":46},"rbd lock list \u003Cpool>\u002F\u003Cimage>\nrbd lock remove \u003Cpool>\u002F\u003Cimage> \u003Clock-id> \u003Clocker>\nceph osd blocklist add \u003Cfailed-node-ip>\n",[14,227,228,262,308],{"__ignoreMap":46},[150,229,230,233,236,239,241,244,247,249,252,255,258,260],{"class":152,"line":153},[150,231,232],{"class":156},"rbd",[150,234,235],{"class":160}," lock",[150,237,238],{"class":160}," list",[150,240,174],{"class":167},[150,242,243],{"class":160},"poo",[150,245,246],{"class":180},"l",[150,248,206],{"class":167},[150,250,251],{"class":160},"\u002F",[150,253,254],{"class":167},"\u003C",[150,256,257],{"class":160},"imag",[150,259,181],{"class":180},[150,261,184],{"class":167},[150,263,264,266,268,271,273,275,277,279,281,283,285,287,289,291,294,296,298,300,303,306],{"class":152,"line":187},[150,265,232],{"class":156},[150,267,235],{"class":160},[150,269,270],{"class":160}," remove",[150,272,174],{"class":167},[150,274,243],{"class":160},[150,276,246],{"class":180},[150,278,206],{"class":167},[150,280,251],{"class":160},[150,282,254],{"class":167},[150,284,257],{"class":160},[150,286,181],{"class":180},[150,288,206],{"class":167},[150,290,174],{"class":167},[150,292,293],{"class":160},"lock-i",[150,295,203],{"class":180},[150,297,206],{"class":167},[150,299,174],{"class":167},[150,301,302],{"class":160},"locke",[150,304,305],{"class":180},"r",[150,307,184],{"class":167},[150,309,311,314,317,320,323,325,328,330],{"class":152,"line":310},3,[150,312,313],{"class":156},"ceph",[150,315,316],{"class":160}," osd",[150,318,319],{"class":160}," blocklist",[150,321,322],{"class":160}," add",[150,324,174],{"class":167},[150,326,327],{"class":160},"failed-node-i",[150,329,11],{"class":180},[150,331,184],{"class":167},[11,333,334,335,340],{},"The ",[51,336,339],{"href":337,"rel":338},"https:\u002F\u002Fgithub.com\u002Fceph\u002Fceph-csi\u002Fissues\u002F4526",[55],"ceph-csi GitHub issue tracker"," has reports of similar race conditions causing Multi-Attach errors, with maintainers recommending checking kubelet logs and reviewing volume identifiers.",[11,342,343,344,347],{},"Sometimes an RBD device stays mapped to a node even with no workload using it. Check its state with ",[14,345,346],{},"rbd showmapped"," and force-unmap it.",[38,349,351],{"className":144,"code":350,"language":146,"meta":46,"style":46},"rbd showmapped | grep \u003Cpv-name>\nrbd unmap \u002Fdev\u002Frbd27 -o force\n",[14,352,353,373],{"__ignoreMap":46},[150,354,355,357,360,362,364,366,369,371],{"class":152,"line":153},[150,356,232],{"class":156},[150,358,359],{"class":160}," showmapped",[150,361,168],{"class":167},[150,363,171],{"class":156},[150,365,174],{"class":167},[150,367,368],{"class":160},"pv-nam",[150,370,181],{"class":180},[150,372,184],{"class":167},[150,374,375,377,380,383,386],{"class":152,"line":187},[150,376,232],{"class":156},[150,378,379],{"class":160}," unmap",[150,381,382],{"class":160}," \u002Fdev\u002Frbd27",[150,384,385],{"class":209}," -o",[150,387,388],{"class":160}," force\n",[11,390,391],{},"Manual recovery works, but it's not something an operator should have to execute every time something fails. A permanent fix needs automated fencing.",[19,393,395],{"id":394},"_4-a-permanent-fix-automated-fencing-with-networkfence-csi-addons","4. A permanent fix: automated fencing with NetworkFence (CSI-Addons)",[11,397,398],{},[26,399],{"alt":400,"src":401},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance\u002Fsection04.webp",[11,403,404,405,410,411,414],{},"As explained in ",[51,406,409],{"href":407,"rel":408},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-31-rook-rbd-network-fencing\u002Fview",[55],"this Rook-focused blog post",", the ",[14,412,413],{},"NetworkFence"," CRD provided by the CSI-Addons Operator can automate blocking a failed node's Ceph client access.",[416,417,418,425,431,434],"ol",{},[419,420,421,422,424],"li",{},"Create a ",[14,423,413],{}," resource specifying the failed node",[419,426,427,428],{},"The CSI Fencing Controller executes the equivalent of ",[14,429,430],{},"ceph osd blocklist add \u003Cnode-address>",[419,432,433],{},"The failed node's Ceph client access is fully blocked, preventing any accidental writes during recovery",[419,435,436,437,440],{},"After recovery, switch to ",[14,438,439],{},"fenceState: Unfenced"," to clear the blocklist",[11,442,443,444,449,450,453,454,456,457,461],{},"Additionally, ",[51,445,448],{"href":446,"rel":447},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-31-rook-rbd-node-loss\u002Fview",[55],"a related post on handling node loss"," suggests shortening Pod ",[14,451,452],{},"tolerationSeconds"," from the default 300 seconds, cutting the lead time between failure detection and eviction. Combining NetworkFence with a shorter ",[14,455,452],{}," gets you close to fully automated recovery without manual intervention. But designing and maintaining that automation still requires real Ceph operations expertise. A managed K3s platform like ",[51,458,460],{"href":76,"rel":459},[55],"Kubo"," builds this kind of storage operations design in from the start.",[11,463,464,465,470],{},"For cases where an unmap gets stuck, ",[51,466,469],{"href":467,"rel":468},"https:\u002F\u002Fblog.nuvotex.de\u002Frook-ceph-unmap-stuck-rbd\u002F",[55],"this practical recovery write-up"," is a useful reference — the need to check unmap state across multiple nodes is easy to overlook in self-hosted environments.",[19,472,474],{"id":473},"_5-a-proxmox-specific-gotcha-ceph-network-splits-that-freeze-io","5. A Proxmox-specific gotcha: Ceph network splits that freeze I\u002FO",[11,476,477],{},[26,478],{"alt":479,"src":480},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance\u002Fsection05.webp",[11,482,483,484,489,490,493],{},"Running your own Ceph cluster on Proxmox introduces another easily-missed issue. In ",[51,485,488],{"href":486,"rel":487},"https:\u002F\u002Fforum.proxmox.com\u002Fposts\u002F857457\u002F",[55],"a case reported on the Proxmox forums",", a 3-node Ceph cluster had its VM I\u002FO freeze after consecutive node reboots during maintenance — even though Corosync cluster communication was perfectly healthy, ",[63,491,492],{},"the dedicated Ceph network interface lost connectivity"," between nodes.",[11,495,496,497,502,503,102],{},"This happens because Corosync's heartbeat and Ceph's public\u002Fcluster network are entirely separate communication paths. ",[51,498,501],{"href":499,"rel":500},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FFull_Mesh_Network_for_Ceph_Server",[55],"Proxmox's official full-mesh networking guide"," recommends isolating the Ceph network from the Proxmox VE cluster LAN using a separate mesh configuration. When planning node maintenance, don't assume \"Corosync is alive, so we're fine\" — separately verify that ",[63,504,505],{},"the Ceph-specific network hasn't split off on its own",[19,507,509],{"id":508},"summary","Summary",[11,511,512],{},"The Multi-Attach error during node maintenance on Proxmox K3s + Ceph CSI (RBD) comes down to a race condition between two asynchronous mechanisms: a lingering RBD exclusive lock and watcher, and the timing gap in VolumeAttachment garbage collection. Knowing the manual recovery steps helps, but adding automated fencing via NetworkFence (CSI-Addons) cuts down manual intervention dramatically.",[11,514,515,516,521],{},"That said, running fencing design, network isolation, and lock management entirely in-house is no small burden. ",[51,517,520],{"href":518,"rel":519},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[55],"Kubo On-Premise"," takes on that complexity, offering fully managed Kubernetes on top of your own infrastructure — Proxmox included — with Air-Gapped support so you can offload storage and fencing operations while keeping full data sovereignty.",[11,523,524,525,528,529,534],{},"For managed K8s in the cloud, ",[51,526,460],{"href":76,"rel":527},[55]," runs on a lightweight K3s-based setup that costs less than EKS or AKS while still delivering full Kubernetes. If you're wrestling with similar operational pain, feel free to ",[51,530,533],{"href":531,"rel":532},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[55],"get in touch"," and talk through your setup.",[536,537,538],"style",{},"html pre.shiki code .sE3pS, html code.shiki .sE3pS{--shiki-default:#C0CAF5}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sGX4V, html code.shiki .sGX4V{--shiki-default:#A9B1D6}html pre.shiki code .sT800, html code.shiki .sT800{--shiki-default:#E0AF68}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":46,"searchDepth":187,"depth":187,"links":540},[541,542,543,544,545,546],{"id":21,"depth":187,"text":22},{"id":82,"depth":187,"text":83},{"id":128,"depth":187,"text":129},{"id":394,"depth":187,"text":395},{"id":473,"depth":187,"text":474},{"id":508,"depth":187,"text":509},"2026-11-14","Draining a node on a Proxmox-hosted K3s cluster left a Ceph CSI (RBD) PV stuck in Multi-Attach error. The cause: a stale exclusive lock, a lingering watcher, and a fencing race condition. Recovery commands and a permanent NetworkFence fix inside.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance",{"title":5,"description":548},"blog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance",[558,559,560,561,232,562],"k3s","kubernetes","proxmox","ceph-csi","storage","TF41vh3x8D049cH-tkTxkNKAybV2tJE4tJhmZZV3s9Y",[565,574,583,593,600,610],{"path":566,"title":567,"description":568,"date":569,"tags":570},"\u002Fblog\u002Fen\u002Fk3s-longhorn-openebs-mayastor-storage-selection","Is Longhorn Enough, or Do You Need OpenEBS Mayastor? Deciding K3s Storage by Performance Requirements","Still defaulting to Longhorn for K3s storage without a second thought? We compare Longhorn and OpenEBS Mayastor across hardware requirements, replica configuration, and performance characteristics, with a checklist for choosing without regret.","2026-11-15",[558,559,571,572,573,562],"longhorn","openebs","mayastor",{"path":575,"title":576,"description":577,"date":578,"tags":579},"\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum","A K3s Cluster Built on 2 Proxmox Hosts Went Down Entirely After a Brief Network Blip. The Culprit Was Corosync's Self-Fencing","A K3s cluster running on 2 Proxmox hosts went down completely after a few seconds of network interruption: both the host and the etcd quorum were taken out together. Here's why Corosync's self-fencing was the real cause, and how node placement design prevents it.","2026-11-11",[558,559,560,580,581,582],"etcd","high-availability","quorum",{"path":584,"title":585,"description":586,"date":587,"tags":588},"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","2026-09-12",[558,559,562,589,590,591,592],"local-path-provisioner","persistent-volume","node-affinity","zfs",{"path":594,"title":595,"description":596,"date":597,"tags":598},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[558,559,560,580,599,581],"live-migration",{"path":601,"title":602,"description":603,"date":604,"tags":605},"\u002Fblog\u002Fen\u002Fk3s-on-proxmox-cluster","Building a K3s Cluster on Proxmox: A Step-by-Step Guide","Complete guide to deploying a lightweight K3s Kubernetes cluster on Proxmox VE, covering VM templates, HA configuration, storage, and Ingress setup.","2026-05-27",[558,559,560,606,607,608,609],"self-hosting","containers","cluster","devops",{"path":611,"title":612,"description":613,"date":614,"tags":615},"\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap","We Couldn't Reconstruct What an AI Agent Did to Our Kubernetes API. The Gap Was in Our Audit Log Settings","After running AI agents on Kubernetes, we realized that when something went wrong, the Audit Log alone couldn't tell us who did what, or why. Here's the Audit Policy blind spot and how we fixed agent traceability.","2026-11-09",[558,559,616,617,618,619],"ai-agent","audit-log","rbac","observability",1794902504507]