[{"data":1,"prerenderedAt":341},["ShallowReactive",2],{"blog-en-proxmox-corosync-self-fencing-k3s-etcd-quorum":3,"blog-related-en-proxmox-corosync-self-fencing-k3s-etcd-quorum":291,"blog-en-proxmox-corosync-self-fencing-k3s-etcd-quorum-alt":279},{"id":4,"title":5,"author":6,"body":7,"date":273,"description":274,"extension":275,"image":276,"locale":277,"meta":278,"navigation":279,"path":280,"seo":281,"stem":282,"tags":283,"__hash__":290},"blog\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum.md","A K3s Cluster Built on 2 Proxmox Hosts Went Down Entirely After a Brief Network Blip. The Culprit Was Corosync's Self-Fencing","Kubo Team",{"type":8,"value":9,"toc":262},"minimark",[10,15,23,26,29,40,44,50,59,66,69,78,82,88,102,118,121,129,135,144,176,185,189,195,202,235,238,242,245],[11,12,14],"h2",{"id":13},"_1-the-incident-a-few-seconds-of-network-downtime-took-down-the-entire-k3s-cluster","1. The Incident — A Few Seconds of Network Downtime Took Down the Entire K3s Cluster",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum\u002Fsection01.webp",[16,24,25],{},"In one environment running a self-managed K3s cluster on Proxmox, this is what happened: a network switch restarted, causing a few seconds of interruption. That should have been it. But when things came back, the K3s API server was completely unreachable.",[16,27,28],{},"On investigation, the K3s server nodes were running a 3-node configuration, deliberately set to an odd number to maintain etcd quorum (majority consensus). That part of the design was correct. But the 3 VMs were distributed across only 2 Proxmox hosts, in a 2-and-1 split. That's where the oversight was.",[16,30,31,32,39],{},"The network blip caused Proxmox's cluster communication layer (Corosync) to lose quorum, and the host running 2 of the VMs was rebooted by the watchdog-based self-fencing mechanism described in ",[33,34,38],"a",{"href":35,"rel":36},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FHigh_Availability",[37],"nofollow","Proxmox VE's official High Availability documentation",". As a result, the 2 K3s server nodes on that host disappeared simultaneously, dropping K3s's own etcd quorum down to a single surviving node out of three — a majority loss.",[11,41,43],{"id":42},"_2-why-we-used-an-odd-number-of-nodes-wasnt-enough","2. Why \"We Used an Odd Number of Nodes\" Wasn't Enough",[16,45,46],{},[19,47],{"alt":48,"src":49},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum\u002Fsection02.webp",[16,51,52,53,58],{},"K3s's embedded etcd is designed around an odd number of server nodes (3 or 5), staying operational as long as a majority remain alive. ",[33,54,57],{"href":55,"rel":56},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-20-k3s-embedded-ha-etcd\u002Fmarkdown",[37],"A guide to configuring K3s with embedded HA (etcd)"," notes that a 3-node setup tolerates 1 failure, and a 5-node setup tolerates 2, with a recommended latency of under 10ms between server nodes.",[16,60,61,62],{},"That design assumes K3s nodes fail independently. But in practice, since they're VMs, K3s nodes always run on top of some physical host — in this case, Proxmox. ",[63,64,65],"strong",{},"K3s's quorum design and Proxmox's quorum design are separate layers, but node placement ties them together.",[16,67,68],{},"When 3 K3s nodes are split 2-to-1 across 2 hosts, losing \"just one host\" — an ordinary-sounding failure — directly becomes \"losing 2 K3s nodes at once\" on the Kubernetes side, a failure mode the odd-node design never anticipated. Using an odd number of nodes was correct in itself; the placement design simply created a different weak point.",[16,70,71,72,77],{},"With a managed K3s platform like ",[33,73,76],{"href":74,"rel":75},"https:\u002F\u002Fkubo.hexabase.io\u002F",[37],"Kubo",", this kind of node placement design spanning both the hypervisor layer and the Kubernetes layer is handled as standard, without operators needing to track it themselves.",[11,79,81],{"id":80},"_3-the-real-culprit-corosyncs-self-fencing-a-structural-weakness-of-2-node-proxmox-clusters","3. The Real Culprit: Corosync's Self-Fencing — A Structural Weakness of 2-Node Proxmox Clusters",[16,83,84],{},[19,85],{"alt":86,"src":87},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum\u002Fsection03.webp",[16,89,90,91,96,97,101],{},"Proxmox's cluster management relies on ",[33,92,95],{"href":93,"rel":94},"https:\u002F\u002Fwww.mankier.com\u002F3\u002Fvotequorum_overview",[37],"Corosync's"," majority-vote quorum. Proxmox VE's official documentation states that cluster-wide locking requires quorum to be formed, and that while quorum is lost, a node cannot reset its watchdog — once the watchdog timer expires, the entire host reboots after 60 seconds (see ",[33,98,100],{"href":35,"rel":99},[37],"Proxmox VE High Availability",").",[16,103,104,105,108,109,112,113,101],{},"This becomes a structural weakness in 2-node configurations. With 3 nodes, a single failure still leaves a majority (2 of 3 votes). With 2 nodes, the moment one becomes unreachable, the remaining node holds only 1 of 2 votes and loses quorum. Multiple reports on Proxmox's community forums describe one node rebooting and the other ",[63,106,107],{},"also rebooting roughly 60 seconds later due to watchdog expiration",", and cases where a broad network interruption — such as a switch restart — caused ",[63,110,111],{},"both nodes to self-fence simultaneously"," (see ",[33,114,117],{"href":115,"rel":116},"https:\u002F\u002Fforum.proxmox.com\u002Fthreads\u002F2-node-cluster-watchdog-expire-reboot.60979\u002F",[37],"Proxmox community forum",[16,119,120],{},"There's an irony here: a safety mechanism built to prevent split-brain can, depending on node placement, become the very trigger that takes down the entire cluster at once.",[11,122,124,125],{"id":123},"_4-recovery-and-immediate-mitigation-qdevice-and-cluster-reset","4. Recovery and Immediate Mitigation — QDevice and ",[126,127,128],"code",{},"cluster-reset",[16,130,131],{},[19,132],{"alt":133,"src":134},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum\u002Fsection04.webp",[16,136,137,138,143],{},"The structural fix on the Proxmox side is introducing a ",[33,139,142],{"href":140,"rel":141},"https:\u002F\u002Fdocumentation.suse.com\u002Fsle-ha\u002F15-SP2\u002Fhtml\u002FSLE-HA-all\u002Fcha-ha-qdevice.html",[37],"QDevice (corosync-qdevice \u002F corosync-qnetd)",". By placing an independent external host outside the 2-node cluster and giving it a third vote, the remaining node can still hold 2 of 2 votes if one host goes down, preserving quorum. This is the standard recommended configuration for even-numbered clusters, and especially for 2-node clusters.",[16,145,146,147,150,151,156,157,160,161,166,167,172,173,175],{},"If K3s's own etcd quorum does end up broken, ",[126,148,149],{},"k3s server --cluster-reset"," is the entry point for recovery. According to ",[33,152,155],{"href":153,"rel":154},"https:\u002F\u002Fdocs.k3s.io\u002Fcli\u002Fserver",[37],"K3s's official server CLI reference",", this flag makes the node \"forget all peers and become sole member of a new cluster\" — a non-destructive reset. There's a catch, though: if the IP used during restoration doesn't match the original ",[126,158,159],{},"--node-ip"," setting, that network mismatch can itself become a new source of instability, as documented in a ",[33,162,165],{"href":163,"rel":164},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F5070",[37],"similar report on K3s's official GitHub repository",". A ",[33,168,171],{"href":169,"rel":170},"https:\u002F\u002Fforums.suse.com\u002Ft\u002Fk3s-w-etcd-fails-to-restart-after-ip-change\u002F42496",[37],"real-world example posted on the SUSE Forums"," confirms that the error \"this server is not a member of the etcd cluster\" stems from exactly this kind of IP mismatch. It's safer to fix ",[126,174,159],{}," permanently in config.yaml rather than specifying it ad hoc on the command line each time.",[16,177,178,179,184],{},"With ",[33,180,183],{"href":181,"rel":182},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[37],"Kubo On-Premise",", recovery procedures for this kind of quorum loss are built into standard operations, so you're not left cross-referencing runbooks and commands in the middle of an incident.",[11,186,188],{"id":187},"_5-the-real-fix-treat-node-placement-and-clustering-design-as-separate-problems","5. The Real Fix — Treat Node Placement and Clustering Design as Separate Problems",[16,190,191],{},[19,192],{"alt":193,"src":194},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum\u002Fsection05.webp",[16,196,197,198,201],{},"The real lesson from this incident isn't just \"2-node clusters have a known weakness,\" as Proxmox's own forums point out — it's that ",[63,199,200],{},"the host-layer quorum design and the Kubernetes-layer quorum design were never examined as two independent problems."," There are three concrete things to get right to prevent a repeat:",[203,204,205,212,223],"ul",{},[206,207,208,211],"li",{},[63,209,210],{},"Spread nodes across hosts",": distribute K3s server nodes one-per-host across at least 3 Proxmox hosts. Don't concentrate them on just 2",[206,213,214,217,218,222],{},[63,215,216],{},"Use a fixed registration address",": put a VIP or load balancer in front of the server node group so the cluster doesn't depend on any single node's IP (see the ",[33,219,221],{"href":55,"rel":220},[37],"K3s embedded HA configuration guide",")",[206,224,225,228,229,234],{},[63,226,227],{},"Understand the timeout design",": etcd's defaults are a 100ms heartbeat interval and a 1000ms election timeout. ",[33,230,233],{"href":231,"rel":232},"https:\u002F\u002Fetcd.io\u002Fdocs\u002Fv3.6\u002Ftuning\u002F",[37],"etcd's official tuning documentation"," recommends setting the election timeout to at least 10 times the round-trip time between members. If network latency eats into that margin, unnecessary leader elections can fire, destabilizing the cluster temporarily",[16,236,237],{},"Both Proxmox's quorum design and K3s's etcd quorum design can be assembled correctly on their own, strictly by following each project's official documentation. The problem is that neither document, on its own, teaches you about the intersection — node placement — where both designs have to hold at the same time.",[11,239,241],{"id":240},"summary","Summary",[16,243,244],{},"Even with an odd number of nodes securing etcd quorum, if you haven't also designed the number and placement of the underlying Proxmox hosts, a hypervisor-layer failure can translate directly into a Kubernetes-layer failure. Proxmox's self-fencing exists to prevent split-brain, but in a 2-node setup with the wrong node placement, that same safety mechanism can become the trigger for the outage it was meant to prevent.",[16,246,247,248,251,252,255,256,261],{},"Running Proxmox and K3s yourself means reconciling host count, whether you need a QDevice, K3s node placement, and timeout design, all on your own. ",[33,249,76],{"href":74,"rel":250},[37]," is a managed Kubernetes platform built on K3s with this kind of availability design already built in, and the same design carries over to on-premises environments through ",[33,253,183],{"href":181,"rel":254},[37],". If you'd like to reduce the complexity of running this kind of setup yourself, you can also reach out via ",[33,257,260],{"href":258,"rel":259},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[37],"our contact page"," to discuss your architecture.",{"title":263,"searchDepth":264,"depth":264,"links":265},"",2,[266,267,268,269,271,272],{"id":13,"depth":264,"text":14},{"id":42,"depth":264,"text":43},{"id":80,"depth":264,"text":81},{"id":123,"depth":264,"text":270},"4. Recovery and Immediate Mitigation — QDevice and cluster-reset",{"id":187,"depth":264,"text":188},{"id":240,"depth":264,"text":241},"2026-11-11","A K3s cluster running on 2 Proxmox hosts went down completely after a few seconds of network interruption: both the host and the etcd quorum were taken out together. Here's why Corosync's self-fencing was the real cause, and how node placement design prevents it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum",{"title":5,"description":274},"blog\u002Fen\u002Fproxmox-corosync-self-fencing-k3s-etcd-quorum",[284,285,286,287,288,289],"k3s","kubernetes","proxmox","etcd","high-availability","quorum","upaqFzBVn6RNUAkS6vB4rE6c7Tb9aAT_94tSrDfAnwg",[292,299,306,314,323,332],{"path":293,"title":294,"description":295,"date":296,"tags":297},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[284,285,286,287,298,288],"live-migration",{"path":300,"title":301,"description":302,"date":303,"tags":304},"\u002Fblog\u002Fen\u002Fk3s-kubeadm-ha-etcd-failover-rto-gap","Your RTO Said 30 Seconds. It Took 3 Minutes. Why K3s and kubeadm HA Specs Don't Match Reality","How long does failover really take with K3s embedded etcd versus kubeadm stacked\u002Fexternal etcd? We examine the gap between official RTO targets and real-world recovery times, and how failure domains and node counts differ.","2026-10-09",[284,285,287,288,305],"kubeadm",{"path":307,"title":308,"description":309,"date":310,"tags":311},"\u002Fblog\u002Fen\u002Fk3s-etcd-snapshot-restore-token-cluster-reset-pitfalls","The Automated etcd Snapshots Were Running Every Day. Here's Why K3s Restore Still Failed in Production","K3s's etcd snapshots looked healthy every day, yet restore failed during a real outage. Three real-world traps — token management, S3 restore incidents, and cluster-reset pitfalls — reveal what's really at risk in embedded etcd backups.","2026-09-21",[284,285,287,312,313,288],"backup-restore","disaster-recovery",{"path":315,"title":316,"description":317,"date":318,"tags":319},"\u002Fblog\u002Fen\u002Fk3s-proxmox-ceph-csi-rbd-multiattach-node-maintenance","Multi-Attach error for volume: What to Know About RBD Locks Before Node Maintenance on Proxmox K3s + Ceph CSI","Draining a node on a Proxmox-hosted K3s cluster left a Ceph CSI (RBD) PV stuck in Multi-Attach error. The cause: a stale exclusive lock, a lingering watcher, and a fencing race condition. Recovery commands and a permanent NetworkFence fix inside.","2026-11-14",[284,285,286,320,321,322],"ceph-csi","rbd","storage",{"path":324,"title":325,"description":326,"date":327,"tags":328},"\u002Fblog\u002Fen\u002Fk3s-vs-microk8s-ha-etcd-dqlite-edge-production","K3s vs MicroK8s: Which One Actually Wins in Production? The HA Fork Between dqlite and etcd","K3s vs MicroK8s: both are called 'lightweight Kubernetes,' but the real production divide is HA architecture. We compare K3s's embedded etcd and MicroK8s's dqlite using official documentation data.","2026-10-25",[284,329,285,288,330,331],"microk8s","edge-computing","comparison",{"path":333,"title":334,"description":335,"date":336,"tags":337},"\u002Fblog\u002Fen\u002Fk3s-rke2-migration-production-downtime-cis-ha","We Migrated K3s to RKE2 and Production Went Down for Two Hours. What \"Lightweight K8s\" Buyers Overlook","K3s vs RKE2 is often reduced to a feature checklist. Using a real production migration incident, we break down CIS Hardening, HA requirements, and storage assumptions to help you choose the right distribution for production.","2026-10-05",[284,338,285,339,288,340],"rke2","cis-benchmark","production-kubernetes",1794902517611]