[{"data":1,"prerenderedAt":470},["ShallowReactive",2],{"blog-en-k3s-longhorn-storage-network-multus-replica-rebuild-stall":3,"blog-related-en-k3s-longhorn-storage-network-multus-replica-rebuild-stall":409,"blog-en-k3s-longhorn-storage-network-multus-replica-rebuild-stall-alt":397},{"id":4,"title":5,"author":6,"body":7,"date":391,"description":392,"extension":393,"image":394,"locale":395,"meta":396,"navigation":397,"path":398,"seo":399,"stem":400,"tags":401,"__hash__":408},"blog\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall.md","We Rebooted a K3s Node and Longhorn's Replica Rebuild Silently Stalled at 0%. The Storage Network Startup Race Nobody Warned Us About","Kubo Team",{"type":8,"value":9,"toc":377},"minimark",[10,15,33,56,63,80,84,98,119,130,136,140,156,163,171,175,178,183,208,214,218,251,258,262,265,318,324,331,338,342,345,358,361],[11,12,14],"h2",{"id":13},"_1-the-symptom-the-ui-says-rebuilding-but-the-replica-never-moves-past-0","1. The Symptom — The UI Says \"Rebuilding,\" But the Replica Never Moves Past 0%",[16,17,18,19,23,24,27,28,32],"p",{},"If you run PersistentVolumes on Longhorn inside a K3s cluster, this scenario may sound familiar.\nYou reboot a node for routine maintenance. A while later you check the Longhorn UI and the affected\nvolume is still ",[20,21,22],"code",{},"degraded",". Open the details and it says ",[20,25,26],{},"rebuilding",", as if progress is happening.\nBut hours later, ",[29,30,31],"strong",{},"the replica rebuild is still stuck at 0%",".",[16,34,35,36,39,40,43,44,51,52,55],{},"This isn't a hypothetical. Harvester, the upstream project that builds on Longhorn, reported exactly\nthis symptom in production: after a node reboot, 38 volumes stayed stuck in \"attached\u002Fdegraded\" state\nfor 37 hours, with ",[20,37,38],{},"rebuildStatus"," reporting ",[20,41,42],{},"isRebuilding: true"," the entire time\n(",[45,46,50],"a",{"href":47,"rel":48},"https:\u002F\u002Fgithub.com\u002Fharvester\u002Fharvester\u002Fissues\u002F11639",[49],"nofollow","Harvester #11639","). The UI insists it's working,\nbut underneath it's stuck in an infinite loop. That's the ",[29,53,54],{},"silent failure"," this article is about.",[16,57,58],{},[59,60],"img",{"alt":61,"src":62},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall\u002Fsection01.webp",[16,64,65,66,69,70,73,74,79],{},"What makes this kind of failure dangerous is that it rarely triggers an alert. The volume is still\n",[20,67,68],{},"attached",", and the Pod keeps running. ",[29,71,72],{},"You've simply lost redundancy",", so nobody notices until the\nnext node goes down. If you run Longhorn in production on K3s, you need a way to verify that replica\nrebuilds are actually progressing, not just reporting a status. Whether you build that monitoring\nyourself or hand it to a managed K3s platform like ",[45,75,78],{"href":76,"rel":77},"https:\u002F\u002Fkubo.hexabase.io\u002F",[49],"Kubo"," that ships\nmonitoring out of the box is a decision worth thinking through.",[11,81,83],{"id":82},"_2-the-cause-instance-manager-started-one-second-before-the-storage-network-was-ready","2. The Cause — instance-manager Started One Second Before the Storage Network Was Ready",[16,85,86,87,90,91,94,97],{},"Digging into the Harvester bug report, the cause comes down to a surprisingly narrow timing gap.\nAfter the node rebooted, Longhorn's ",[20,88,89],{},"instance-manager"," pod came up ",[29,92,93],{},"just one second before",[20,95,96],{},"rke2-multus",", the component providing the dedicated storage network, finished getting ready.\nThat one-second gap turned out to be fatal.",[16,99,100,101,104,105,110,111,114,115,118],{},"In production, Longhorn offers a ",[29,102,103],{},"Storage Network"," feature to isolate replication traffic from the\ncluster's regular CNI network. Without that isolation, replication traffic competes for bandwidth with\nevery other Pod on the node, making latency unpredictable\n(",[45,106,109],{"href":107,"rel":108},"https:\u002F\u002Flonghorn.io\u002Fdocs\u002F1.12.1\u002Fadvanced-resources\u002Fdeploy\u002Fstorage-network\u002F",[49],"Longhorn official docs: Storage Network",").\nIt's implemented via Multus attaching a second network interface, and enabling it causes Longhorn's\ninstance-manager, backing-image-manager, and backing-image-data-source pods to be ",[29,112,113],{},"recreated"," with a\n",[20,116,117],{},"k8s.v1.cni.cncf.io\u002Fnetworks"," annotation attached.",[16,120,121,122,125,126,129],{},"In the one-second window, instance-manager requested a storage network attachment but hadn't yet\nreceived the ",[20,123,124],{},"k8s.v1.cni.cncf.io\u002Fnetworks-status"," annotation from Multus. Longhorn's implementation\nfalls back to the pod network IP whenever that annotation is missing, which meant ",[29,127,128],{},"the source engine\nwas on the storage network while the target replica was only reachable over the pod network IP"," — a\nrouting mismatch. That mismatch is the direct reason the replica rebuild never advanced past 0%.",[16,131,132],{},[59,133],{"alt":134,"src":135},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall\u002Fsection02.webp",[11,137,139],{"id":138},"_3-why-that-one-second-is-critical-a-multus-version-change-made-it-worse","3. Why That One Second Is Critical — A Multus Version Change Made It Worse",[16,141,142,143,145,146,149,150,155],{},"What makes this even trickier is that it isn't just a one-off timing race. Longhorn has long relied on\nthe ",[20,144,124],{}," annotation to check the state of the storage network. But\nMultus deprecated that annotation in v3.6 and ",[29,147,148],{},"removed it entirely in v4.0.0","\n(",[45,151,154],{"href":152,"rel":153},"https:\u002F\u002Fgithub.com\u002Flonghorn\u002Flonghorn\u002Fissues\u002F6953",[49],"longhorn\u002Flonghorn #6953","). On any cluster running\nMultus v4.0.0 or later, Longhorn can no longer recognize the storage network's state at all — no\ntiming race required.",[16,157,158,159,162],{},"In other words, the same \"replica rebuild won't progress\" symptom can come from ",[29,160,161],{},"two entirely\ndifferent root causes",": ① a one-time startup timing race, or ② a permanent incompatibility introduced\nby a Multus version upgrade. If you're running Storage Network with Multus on K3s, you need to verify\nthe Multus and Longhorn version combination before you rely on it.",[16,164,165,166,170],{},"Chasing this kind of cross-component version compatibility yourself takes real, ongoing effort.\n",[45,167,169],{"href":76,"rel":168},[49],"Kubo Cloud"," ships with CNI and storage components validated together as a\ntested combination, which reduces the \"it broke after we upgraded\" class of incident.",[11,172,174],{"id":173},"_4-the-design-gap-everyone-misses-replica-count-and-the-last-healthy-replica-drain-protection","4. The Design Gap Everyone Misses — Replica Count and the \"Last Healthy Replica\" Drain Protection",[16,176,177],{},"Even if you dodge the startup race and the network mismatch, the replica design itself hides another\npitfall.",[179,180,182],"h3",{"id":181},"should-replica-count-be-2-or-3","Should Replica Count Be 2 or 3?",[16,184,185,186,189,190,195,196,149,199,204,205,32],{},"Longhorn defaults to a replica count of 3, but production tuning guides also suggest ",[29,187,188],{},"lowering the\ndefault to 2"," to cut disk usage and improve write latency\n(",[45,191,194],{"href":192,"rel":193},"https:\u002F\u002Fcloudcasa.io\u002Fblog\u002Flonghorn-on-production-clusters-storage-configuration-tuning-and-gotchas\u002F",[49],"CloudCasa: Longhorn on Production Clusters",").\nOn the other hand, a separate best-practices guide notes that a replica count of 2 tolerates only one\nnode failure, while a replica count of 3 tolerates two, and ",[29,197,198],{},"recommends 3 for production",[45,200,203],{"href":201,"rel":202},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-20-set-longhorn-volume-replica-count\u002Fview",[49],"OneUptime: How to Set Longhorn Volume Replica Count",").\nWhich is \"correct\" depends on the workload's criticality, but ",[29,206,207],{},"plenty of clusters run on the default\nvalue without anyone having weighed that trade-off",[16,209,210],{},[59,211],{"alt":212,"src":213},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall\u002Fsection03.webp",[179,215,217],{"id":216},"when-node-drain-wont-finish","When Node Drain Won't Finish",[16,219,220,221,149,224,229,230,233,234,239,240,243,244,246,247,250],{},"The other pitfall shows up during node maintenance. Cordoning a node makes Longhorn automatically stop\nscheduling new workloads onto it, but ",[29,222,223],{},"automatic replica eviction on cordon is not implemented",[45,225,228],{"href":226,"rel":227},"https:\u002F\u002Fgithub.com\u002Flonghorn\u002Flonghorn\u002Fdiscussions\u002F5736",[49],"longhorn\u002Flonghorn Discussion #5736","). When you\nthen run drain, if the replica on that node happens to be the ",[29,231,232],{},"last healthy replica"," for a volume,\nLonghorn blocks the drain from completing at all, as a data-protection measure\n(",[45,235,238],{"href":236,"rel":237},"https:\u002F\u002Flonghorn.io\u002Fdocs\u002F1.12.1\u002Fmaintenance\u002Fmaintenance\u002F",[49],"Longhorn Node Maintenance Guide","). At that\npoint, ",[20,241,242],{},"kubectl drain"," will hang indefinitely unless you manually evict the replica from the UI or\nchange the Node Drain Policy. And ",[20,245,242],{}," itself requires ",[20,248,249],{},"--ignore-daemonsets"," — without it,\nLonghorn's manager, CSI plugin, and engine image DaemonSets will block the drain from even starting.",[16,252,253,254,257],{},"Writing this maintenance procedure down so the whole team follows the same steps is unglamorous but\nessential work. With ",[45,255,169],{"href":76,"rel":256},[49],"'s Captain UI, replica placement per volume\nis visualized, making it easier to check ahead of time which node is holding the last healthy replica\nfor any given volume.",[11,259,261],{"id":260},"_5-a-production-checklist-for-running-longhorn-safely-on-k3s","5. A Production Checklist for Running Longhorn Safely on K3s",[16,263,264],{},"Given these failure patterns, here's what to verify before running K3s + Longhorn in production.",[266,267,268,279,295,301,307],"ul",{},[269,270,271,274,275,278],"li",{},[29,272,273],{},"Monitor actual rebuild progress",": don't rely on ",[20,276,277],{},"isRebuilding"," alone — alert on the real progress percentage",[269,280,281,284,285,288,289,294],{},[29,282,283],{},"Check your Multus version before enabling Storage Network",": assume the ",[20,286,287],{},"networks-status"," annotation is gone on v4.0.0 and above (",[45,290,293],{"href":291,"rel":292},"https:\u002F\u002Flonghorn.io\u002Fdocs\u002F1.12.1\u002Fadvanced-resources\u002Frebuilding\u002F",[49],"Longhorn Rebuilding docs",")",[269,296,297,300],{},[29,298,299],{},"Decide replica count based on workload criticality",", not the default value",[269,302,303,306],{},[29,304,305],{},"Audit \"last healthy replica\" status before draining",": know in advance which volumes will need manual eviction",[269,308,309,312,313,294],{},[29,310,311],{},"Set a snapshot retention policy",": unbounded snapshot growth can lead to disk exhaustion and cascading failures (",[45,314,317],{"href":315,"rel":316},"https:\u002F\u002Fperun.au\u002Finsights\u002Flonghorn-production",[49],"Perun Engineering: Longhorn in production, five failure patterns",[16,319,320],{},[59,321],{"alt":322,"src":323},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall\u002Fsection04.webp",[16,325,326,327,330],{},"None of these items is individually hard to configure. The failure only shows up when ",[29,328,329],{},"node startup\norder, CNI version, replica design, and maintenance procedure"," intersect. Looking at any single layer\nin isolation, you'll never reproduce it — or prevent it.",[16,332,333,334,337],{},"Is manually re-auditing network isolation and replica design across every cluster really the work your\ninfrastructure team wants to be doing? ",[45,335,169],{"href":76,"rel":336},[49]," is K3s-based with a\nRancher management layer and Prometheus\u002FGrafana monitoring built in, making it easier to catch this\nclass of storage-layer anomaly early from a visualized dashboard.",[11,339,341],{"id":340},"summary","Summary",[16,343,344],{},"A Longhorn replica rebuild stuck at 0% isn't a single misconfiguration — it's \"node startup order,\"\n\"a Multus version change,\" \"replica count design,\" and \"drain-time protection\" all compounding into one\nfailure. On a managed offering like EKS or AKS, this kind of cross-component startup ordering and\nversion compatibility is the cloud provider's responsibility. On a self-hosted K3s + Longhorn stack,\nyou're the one chasing all of it.",[16,346,347,348,351,352,357],{},"If you want K3s's lightweight footprint without taking on that operational complexity yourself,\n",[45,349,78],{"href":76,"rel":350},[49]," offers a fully managed Kubernetes environment starting at ¥48,000\u002Fmonth.\nIf you'd like to offload storage design along with the rest of the operational load,\n",[45,353,356],{"href":354,"rel":355},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[49],"reach out to us"," and we'll talk through it.",[16,359,360],{},"Related reading:",[266,362,363,370],{},[269,364,365],{},[45,366,369],{"href":367,"rel":368},"https:\u002F\u002Fkubo.hexabase.io\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending",[49],"A K3s PVC That Never Left Pending: Local Storage That Ignores Capacity and the Node Affinity Trap",[269,371,372],{},[45,373,376],{"href":374,"rel":375},"https:\u002F\u002Fkubo.hexabase.io\u002Fblog\u002Fen\u002Fk3s-vpa-poddisruptionbudget-mass-eviction",[49],"We Turned On Automatic Resource Optimization on K3s and Every Replica Evicted at Once: The Role of PodDisruptionBudget",{"title":378,"searchDepth":379,"depth":379,"links":380},"",2,[381,382,383,384,389,390],{"id":13,"depth":379,"text":14},{"id":82,"depth":379,"text":83},{"id":138,"depth":379,"text":139},{"id":173,"depth":379,"text":174,"children":385},[386,388],{"id":181,"depth":387,"text":182},3,{"id":216,"depth":387,"text":217},{"id":260,"depth":379,"text":261},{"id":340,"depth":379,"text":341},"2026-09-23","A K3s node reboot left a Longhorn replica rebuild stuck at 0%. Based on a real bug report, we break down the Storage Network (Multus) startup race, replica count design, and node drain pitfalls for running Longhorn safely in production.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall",{"title":5,"description":392},"blog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall",[402,403,404,405,406,407],"k3s","kubernetes","longhorn","storage-network","multus","persistent-storage","pjHy6cy4SRP-hKn076cQ3-XBVaxWJ15r54HE6KRo8VM",[410,420,432,442,452,460],{"path":411,"title":412,"description":413,"date":414,"tags":415},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[402,403,416,417,418,419],"hpa","containerresource","autoscaling","service-mesh",{"path":421,"title":422,"description":423,"date":424,"tags":425},"\u002Fblog\u002Fen\u002Fk3s-vs-kubernetes-cni-flannel-default-comparison","K3s vs Standard Kubernetes: How Different Their CNI Starting Points Really Are","K3s ships with Flannel built in, while standard Kubernetes starts with no CNI selected at all. This comparison, grounded in the official K3s, Kubernetes, Calico, and Cilium docs, explains how that difference affects NetworkPolicy enforcement and network performance.","2026-10-20",[402,403,426,427,428,429,430,431],"cni","flannel","calico","cilium","networkpolicy","networking",{"path":433,"title":434,"description":435,"date":436,"tags":437},"\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending","We Drained a Node and a Pod Never Came Back. Why Topology Spread Constraints and PodDisruptionBudget Collide on K3s","Draining a node in production K3s left a Pod stuck Pending. The cause: Topology Spread Constraints' maxSkew colliding with PodDisruptionBudget. We verify how draining itself creates a node shortage, based on official documentation.","2026-10-19",[402,403,438,439,440,441],"topology-spread-constraints","poddisruptionbudget","scheduling","node-drain",{"path":443,"title":444,"description":445,"date":446,"tags":447},"\u002Fblog\u002Fen\u002Fkyverno-validatingadmissionpolicy-cel-migration-silent-bypass","Is the CEL Rewrite of Your Policy Really the Same? How Migrating from Kyverno to ValidatingAdmissionPolicy Let Some Resources Silently Slip Through","As Kyverno's ClusterPolicy heads toward deprecation, teams migrating to ValidatingAdmissionPolicy (CEL) have found that validation silently skipped for certain resource types. Here's the CEL runtime error and failurePolicy trap behind it, and a safe migration path.","2026-10-18",[402,403,448,449,450,451],"validatingadmissionpolicy","kyverno","cel","admission-control",{"path":453,"title":454,"description":455,"date":456,"tags":457},"\u002Fblog\u002Fen\u002Fk3s-vpa-inplaceorrecreate-pod-resize-cgroup","VPA InPlaceOrRecreate Didn't Stop the Restart. Testing InPlacePodVerticalScaling on K3s","Does VPA's InPlaceOrRecreate mode and In-Place Pod Resize really avoid Pod restarts? We test QoS class constraints and the cgroup v1 trap on K3s with real configs and kubectl commands.","2026-10-17",[402,403,458,418,459],"vpa","cgroup",{"path":461,"title":462,"description":463,"date":464,"tags":465},"\u002Fblog\u002Fen\u002Fk3s-pod-security-admission-restricted-servicelb-outage","We Enabled Restricted and ServiceLB Died: The K3s Pod Security Admission Gap No Generic Guide Mentions","Enabling the restricted Pod Security Admission profile on K3s following standard Kubernetes guides can silently kill ServiceLB. Here's why, backed by K3s's own CIS hardening guide, and how to roll it out safely.","2026-10-16",[402,403,466,467,468,469],"pod-security-admission","pod-security-standards","servicelb","security",1792592637654]