Skip to main content

Multi-Attach error for volume: What to Know About RBD Locks Before Node Maintenance on Proxmox K3s + Ceph CSI

Multi-Attach error for volume. If you run a self-hosted K3s cluster on Proxmox with Ceph CSI (RBD) as persistent storage, you've probably seen this message and felt your stomach drop. A routine, planned node maintenance, and suddenly a Pod won't come back from Pending. This article breaks down the cause, the manual recovery steps, and a permanent fix.

1. Right after draining a node, a Pod got stuck in "ContainerCreating"

section01

A kernel update and some Proxmox host maintenance called for draining one node in the K3s cluster — kubectl drain, same as always. But right after the Pod got rescheduled onto another node, this line showed up in the event log:

text
Warning  FailedAttachVolume  pod/app-xxxx  Multi-Attach error for volume "pvc-xxxx" Volume is already exclusively attached to one node and can't be attached to another

Per the official Kubernetes documentation, ReadWriteOnce (RWO) means "mountable read-write by a single node." Multiple Pods on the same node can share it fine, but it can't be attached to two different nodes at once. ReadWriteMany (RWX, supported by CephFS and similar) allows concurrent mounts across nodes — so this issue is specific to block storage like RBD running in RWO mode.

The Pod on the drained node is long gone. So why can't the new node grab the volume? The answer lies in Ceph's own locking mechanism. That kind of fencing design is something every team running a self-hosted K3s cluster eventually has to confront — and it's one of the things worth weighing when evaluating a managed Kubernetes option instead.

2. The cause: a lingering RBD "exclusive lock" and watcher, plus a Kubernetes race condition

section02

Ceph RBD images have the exclusive-lock feature enabled by default. It guarantees that "only a single client can write at any given time," and under normal process termination, the official Ceph documentation explains that the lock transitions cooperatively between clients.

The trouble starts when a node doesn't terminate gracefully — a drain that times out, a kubelet that stops responding, a forced host reboot. In those cases, the old client disappears without releasing its lock. When a new client tries to acquire it, Ceph blocklists the old client and forcibly breaks the lock — but if Kubernetes reschedules the Pod onto another node before that blocklisting completes, the CSI driver tries to attach while the lock is still technically held, and you get a Multi-Attach error.

There's a second, nastier layer: the timing of VolumeAttachment object garbage collection. The official Rook Ceph CSI troubleshooting guide notes that VolumeAttachment objects can linger even after communication with a failed node is lost. When the scheduler tries to re-attach on a new node before Ceph's actual fencing has completed, that asynchronous timing gap is the direct trigger for Multi-Attach errors.

3. Manual recovery: from deleting the VolumeAttachment to releasing the lock

section03

Only if you're certain the failed node is completely down, the following manual recovery steps apply. First check for a lingering VolumeAttachment, then force-delete the Pod if it's stuck Terminating on the old node.

bash
kubectl get volumeattachment | grep <pvc-name>
kubectl delete pod <stuck-pod> --grace-period=0 --force

As Red Hat's support documentation also reports, an RWO ceph-rbd volume can remain unmountable by a new Pod even with no running Pod attached to it. Next, check Ceph's lock state directly and force-release it.

bash
rbd lock list <pool>/<image>
rbd lock remove <pool>/<image> <lock-id> <locker>
ceph osd blocklist add <failed-node-ip>

The ceph-csi GitHub issue tracker has reports of similar race conditions causing Multi-Attach errors, with maintainers recommending checking kubelet logs and reviewing volume identifiers.

Sometimes an RBD device stays mapped to a node even with no workload using it. Check its state with rbd showmapped and force-unmap it.

bash
rbd showmapped | grep <pv-name>
rbd unmap /dev/rbd27 -o force

Manual recovery works, but it's not something an operator should have to execute every time something fails. A permanent fix needs automated fencing.

4. A permanent fix: automated fencing with NetworkFence (CSI-Addons)

section04

As explained in this Rook-focused blog post, the NetworkFence CRD provided by the CSI-Addons Operator can automate blocking a failed node's Ceph client access.

  1. Create a NetworkFence resource specifying the failed node
  2. The CSI Fencing Controller executes the equivalent of ceph osd blocklist add <node-address>
  3. The failed node's Ceph client access is fully blocked, preventing any accidental writes during recovery
  4. After recovery, switch to fenceState: Unfenced to clear the blocklist

Additionally, a related post on handling node loss suggests shortening Pod tolerationSeconds from the default 300 seconds, cutting the lead time between failure detection and eviction. Combining NetworkFence with a shorter tolerationSeconds gets you close to fully automated recovery without manual intervention. But designing and maintaining that automation still requires real Ceph operations expertise. A managed K3s platform like Kubo builds this kind of storage operations design in from the start.

For cases where an unmap gets stuck, this practical recovery write-up is a useful reference — the need to check unmap state across multiple nodes is easy to overlook in self-hosted environments.

5. A Proxmox-specific gotcha: Ceph network splits that freeze I/O

section05

Running your own Ceph cluster on Proxmox introduces another easily-missed issue. In a case reported on the Proxmox forums, a 3-node Ceph cluster had its VM I/O freeze after consecutive node reboots during maintenance — even though Corosync cluster communication was perfectly healthy, the dedicated Ceph network interface lost connectivity between nodes.

This happens because Corosync's heartbeat and Ceph's public/cluster network are entirely separate communication paths. Proxmox's official full-mesh networking guide recommends isolating the Ceph network from the Proxmox VE cluster LAN using a separate mesh configuration. When planning node maintenance, don't assume "Corosync is alive, so we're fine" — separately verify that the Ceph-specific network hasn't split off on its own.

Summary

The Multi-Attach error during node maintenance on Proxmox K3s + Ceph CSI (RBD) comes down to a race condition between two asynchronous mechanisms: a lingering RBD exclusive lock and watcher, and the timing gap in VolumeAttachment garbage collection. Knowing the manual recovery steps helps, but adding automated fencing via NetworkFence (CSI-Addons) cuts down manual intervention dramatically.

That said, running fencing design, network isolation, and lock management entirely in-house is no small burden. Kubo On-Premise takes on that complexity, offering fully managed Kubernetes on top of your own infrastructure — Proxmox included — with Air-Gapped support so you can offload storage and fencing operations while keeping full data sovereignty.

For managed K8s in the cloud, Kubo runs on a lightweight K3s-based setup that costs less than EKS or AKS while still delivering full Kubernetes. If you're wrestling with similar operational pain, feel free to get in touch and talk through your setup.

Related articles

← Back to all posts