Multi-Attach error for volume. If you run a self-hosted K3s cluster on Proxmox with Ceph CSI (RBD) as persistent storage, you've probably seen this message and felt your stomach drop. A routine, planned node maintenance, and suddenly a Pod won't come back from Pending. This article breaks down the cause, the manual recovery steps, and a permanent fix.
1. Right after draining a node, a Pod got stuck in "ContainerCreating"

A kernel update and some Proxmox host maintenance called for draining one node in the K3s cluster — kubectl drain, same as always. But right after the Pod got rescheduled onto another node, this line showed up in the event log:
Warning FailedAttachVolume pod/app-xxxx Multi-Attach error for volume "pvc-xxxx" Volume is already exclusively attached to one node and can't be attached to another
Per the official Kubernetes documentation, ReadWriteOnce (RWO) means "mountable read-write by a single node." Multiple Pods on the same node can share it fine, but it can't be attached to two different nodes at once. ReadWriteMany (RWX, supported by CephFS and similar) allows concurrent mounts across nodes — so this issue is specific to block storage like RBD running in RWO mode.
The Pod on the drained node is long gone. So why can't the new node grab the volume? The answer lies in Ceph's own locking mechanism. That kind of fencing design is something every team running a self-hosted K3s cluster eventually has to confront — and it's one of the things worth weighing when evaluating a managed Kubernetes option instead.
2. The cause: a lingering RBD "exclusive lock" and watcher, plus a Kubernetes race condition

Ceph RBD images have the exclusive-lock feature enabled by default. It guarantees that "only a single client can write at any given time," and under normal process termination, the official Ceph documentation explains that the lock transitions cooperatively between clients.
The trouble starts when a node doesn't terminate gracefully — a drain that times out, a kubelet that stops responding, a forced host reboot. In those cases, the old client disappears without releasing its lock. When a new client tries to acquire it, Ceph blocklists the old client and forcibly breaks the lock — but if Kubernetes reschedules the Pod onto another node before that blocklisting completes, the CSI driver tries to attach while the lock is still technically held, and you get a Multi-Attach error.
There's a second, nastier layer: the timing of VolumeAttachment object garbage collection. The official Rook Ceph CSI troubleshooting guide notes that VolumeAttachment objects can linger even after communication with a failed node is lost. When the scheduler tries to re-attach on a new node before Ceph's actual fencing has completed, that asynchronous timing gap is the direct trigger for Multi-Attach errors.
3. Manual recovery: from deleting the VolumeAttachment to releasing the lock

Only if you're certain the failed node is completely down, the following manual recovery steps apply. First check for a lingering VolumeAttachment, then force-delete the Pod if it's stuck Terminating on the old node.
kubectl get volumeattachment | grep <pvc-name>
kubectl delete pod <stuck-pod> --grace-period=0 --force
As Red Hat's support documentation also reports, an RWO ceph-rbd volume can remain unmountable by a new Pod even with no running Pod attached to it. Next, check Ceph's lock state directly and force-release it.
rbd lock list <pool>/<image>
rbd lock remove <pool>/<image> <lock-id> <locker>
ceph osd blocklist add <failed-node-ip>
The ceph-csi GitHub issue tracker has reports of similar race conditions causing Multi-Attach errors, with maintainers recommending checking kubelet logs and reviewing volume identifiers.
Sometimes an RBD device stays mapped to a node even with no workload using it. Check its state with rbd showmapped and force-unmap it.
rbd showmapped | grep <pv-name>
rbd unmap /dev/rbd27 -o force
Manual recovery works, but it's not something an operator should have to execute every time something fails. A permanent fix needs automated fencing.
4. A permanent fix: automated fencing with NetworkFence (CSI-Addons)

As explained in this Rook-focused blog post, the NetworkFence CRD provided by the CSI-Addons Operator can automate blocking a failed node's Ceph client access.
- Create a
NetworkFenceresource specifying the failed node - The CSI Fencing Controller executes the equivalent of
ceph osd blocklist add <node-address> - The failed node's Ceph client access is fully blocked, preventing any accidental writes during recovery
- After recovery, switch to
fenceState: Unfencedto clear the blocklist
Additionally, a related post on handling node loss suggests shortening Pod tolerationSeconds from the default 300 seconds, cutting the lead time between failure detection and eviction. Combining NetworkFence with a shorter tolerationSeconds gets you close to fully automated recovery without manual intervention. But designing and maintaining that automation still requires real Ceph operations expertise. A managed K3s platform like Kubo builds this kind of storage operations design in from the start.
For cases where an unmap gets stuck, this practical recovery write-up is a useful reference — the need to check unmap state across multiple nodes is easy to overlook in self-hosted environments.
5. A Proxmox-specific gotcha: Ceph network splits that freeze I/O

Running your own Ceph cluster on Proxmox introduces another easily-missed issue. In a case reported on the Proxmox forums, a 3-node Ceph cluster had its VM I/O freeze after consecutive node reboots during maintenance — even though Corosync cluster communication was perfectly healthy, the dedicated Ceph network interface lost connectivity between nodes.
This happens because Corosync's heartbeat and Ceph's public/cluster network are entirely separate communication paths. Proxmox's official full-mesh networking guide recommends isolating the Ceph network from the Proxmox VE cluster LAN using a separate mesh configuration. When planning node maintenance, don't assume "Corosync is alive, so we're fine" — separately verify that the Ceph-specific network hasn't split off on its own.
Summary
The Multi-Attach error during node maintenance on Proxmox K3s + Ceph CSI (RBD) comes down to a race condition between two asynchronous mechanisms: a lingering RBD exclusive lock and watcher, and the timing gap in VolumeAttachment garbage collection. Knowing the manual recovery steps helps, but adding automated fencing via NetworkFence (CSI-Addons) cuts down manual intervention dramatically.
That said, running fencing design, network isolation, and lock management entirely in-house is no small burden. Kubo On-Premise takes on that complexity, offering fully managed Kubernetes on top of your own infrastructure — Proxmox included — with Air-Gapped support so you can offload storage and fencing operations while keeping full data sovereignty.
For managed K8s in the cloud, Kubo runs on a lightweight K3s-based setup that costs less than EKS or AKS while still delivering full Kubernetes. If you're wrestling with similar operational pain, feel free to get in touch and talk through your setup.