1. The Symptom — The UI Says "Rebuilding," But the Replica Never Moves Past 0%
If you run PersistentVolumes on Longhorn inside a K3s cluster, this scenario may sound familiar.
You reboot a node for routine maintenance. A while later you check the Longhorn UI and the affected
volume is still degraded. Open the details and it says rebuilding, as if progress is happening.
But hours later, the replica rebuild is still stuck at 0%.
This isn't a hypothetical. Harvester, the upstream project that builds on Longhorn, reported exactly
this symptom in production: after a node reboot, 38 volumes stayed stuck in "attached/degraded" state
for 37 hours, with rebuildStatus reporting isRebuilding: true the entire time
(Harvester #11639). The UI insists it's working,
but underneath it's stuck in an infinite loop. That's the silent failure this article is about.
What makes this kind of failure dangerous is that it rarely triggers an alert. The volume is still
attached, and the Pod keeps running. You've simply lost redundancy, so nobody notices until the
next node goes down. If you run Longhorn in production on K3s, you need a way to verify that replica
rebuilds are actually progressing, not just reporting a status. Whether you build that monitoring
yourself or hand it to a managed K3s platform like Kubo that ships
monitoring out of the box is a decision worth thinking through.
2. The Cause — instance-manager Started One Second Before the Storage Network Was Ready
Digging into the Harvester bug report, the cause comes down to a surprisingly narrow timing gap.
After the node rebooted, Longhorn's instance-manager pod came up just one second beforerke2-multus, the component providing the dedicated storage network, finished getting ready.
That one-second gap turned out to be fatal.
In production, Longhorn offers a Storage Network feature to isolate replication traffic from the
cluster's regular CNI network. Without that isolation, replication traffic competes for bandwidth with
every other Pod on the node, making latency unpredictable
(Longhorn official docs: Storage Network).
It's implemented via Multus attaching a second network interface, and enabling it causes Longhorn's
instance-manager, backing-image-manager, and backing-image-data-source pods to be recreated with a
k8s.v1.cni.cncf.io/networks annotation attached.
In the one-second window, instance-manager requested a storage network attachment but hadn't yet
received the k8s.v1.cni.cncf.io/networks-status annotation from Multus. Longhorn's implementation
falls back to the pod network IP whenever that annotation is missing, which meant the source engine
was on the storage network while the target replica was only reachable over the pod network IP — a
routing mismatch. That mismatch is the direct reason the replica rebuild never advanced past 0%.
3. Why That One Second Is Critical — A Multus Version Change Made It Worse
What makes this even trickier is that it isn't just a one-off timing race. Longhorn has long relied on
the k8s.v1.cni.cncf.io/networks-status annotation to check the state of the storage network. But
Multus deprecated that annotation in v3.6 and removed it entirely in v4.0.0
(longhorn/longhorn #6953). On any cluster running
Multus v4.0.0 or later, Longhorn can no longer recognize the storage network's state at all — no
timing race required.
In other words, the same "replica rebuild won't progress" symptom can come from two entirely different root causes: ① a one-time startup timing race, or ② a permanent incompatibility introduced by a Multus version upgrade. If you're running Storage Network with Multus on K3s, you need to verify the Multus and Longhorn version combination before you rely on it.
Chasing this kind of cross-component version compatibility yourself takes real, ongoing effort. Kubo Cloud ships with CNI and storage components validated together as a tested combination, which reduces the "it broke after we upgraded" class of incident.
4. The Design Gap Everyone Misses — Replica Count and the "Last Healthy Replica" Drain Protection
Even if you dodge the startup race and the network mismatch, the replica design itself hides another pitfall.
Should Replica Count Be 2 or 3?
Longhorn defaults to a replica count of 3, but production tuning guides also suggest lowering the default to 2 to cut disk usage and improve write latency (CloudCasa: Longhorn on Production Clusters). On the other hand, a separate best-practices guide notes that a replica count of 2 tolerates only one node failure, while a replica count of 3 tolerates two, and recommends 3 for production (OneUptime: How to Set Longhorn Volume Replica Count). Which is "correct" depends on the workload's criticality, but plenty of clusters run on the default value without anyone having weighed that trade-off.
When Node Drain Won't Finish
The other pitfall shows up during node maintenance. Cordoning a node makes Longhorn automatically stop
scheduling new workloads onto it, but automatic replica eviction on cordon is not implemented
(longhorn/longhorn Discussion #5736). When you
then run drain, if the replica on that node happens to be the last healthy replica for a volume,
Longhorn blocks the drain from completing at all, as a data-protection measure
(Longhorn Node Maintenance Guide). At that
point, kubectl drain will hang indefinitely unless you manually evict the replica from the UI or
change the Node Drain Policy. And kubectl drain itself requires --ignore-daemonsets — without it,
Longhorn's manager, CSI plugin, and engine image DaemonSets will block the drain from even starting.
Writing this maintenance procedure down so the whole team follows the same steps is unglamorous but essential work. With Kubo Cloud's Captain UI, replica placement per volume is visualized, making it easier to check ahead of time which node is holding the last healthy replica for any given volume.
5. A Production Checklist for Running Longhorn Safely on K3s
Given these failure patterns, here's what to verify before running K3s + Longhorn in production.
- Monitor actual rebuild progress: don't rely on
isRebuildingalone — alert on the real progress percentage - Check your Multus version before enabling Storage Network: assume the
networks-statusannotation is gone on v4.0.0 and above (Longhorn Rebuilding docs) - Decide replica count based on workload criticality, not the default value
- Audit "last healthy replica" status before draining: know in advance which volumes will need manual eviction
- Set a snapshot retention policy: unbounded snapshot growth can lead to disk exhaustion and cascading failures (Perun Engineering: Longhorn in production, five failure patterns)
None of these items is individually hard to configure. The failure only shows up when node startup order, CNI version, replica design, and maintenance procedure intersect. Looking at any single layer in isolation, you'll never reproduce it — or prevent it.
Is manually re-auditing network isolation and replica design across every cluster really the work your infrastructure team wants to be doing? Kubo Cloud is K3s-based with a Rancher management layer and Prometheus/Grafana monitoring built in, making it easier to catch this class of storage-layer anomaly early from a visualized dashboard.
Summary
A Longhorn replica rebuild stuck at 0% isn't a single misconfiguration — it's "node startup order," "a Multus version change," "replica count design," and "drain-time protection" all compounding into one failure. On a managed offering like EKS or AKS, this kind of cross-component startup ordering and version compatibility is the cloud provider's responsibility. On a self-hosted K3s + Longhorn stack, you're the one chasing all of it.
If you want K3s's lightweight footprint without taking on that operational complexity yourself, Kubo offers a fully managed Kubernetes environment starting at ¥48,000/month. If you'd like to offload storage design along with the rest of the operational load, reach out to us and we'll talk through it.
Related reading: