Skip to main content

A K3s Cluster Built on 2 Proxmox Hosts Went Down Entirely After a Brief Network Blip. The Culprit Was Corosync's Self-Fencing

1. The Incident — A Few Seconds of Network Downtime Took Down the Entire K3s Cluster

section01

In one environment running a self-managed K3s cluster on Proxmox, this is what happened: a network switch restarted, causing a few seconds of interruption. That should have been it. But when things came back, the K3s API server was completely unreachable.

On investigation, the K3s server nodes were running a 3-node configuration, deliberately set to an odd number to maintain etcd quorum (majority consensus). That part of the design was correct. But the 3 VMs were distributed across only 2 Proxmox hosts, in a 2-and-1 split. That's where the oversight was.

The network blip caused Proxmox's cluster communication layer (Corosync) to lose quorum, and the host running 2 of the VMs was rebooted by the watchdog-based self-fencing mechanism described in Proxmox VE's official High Availability documentation. As a result, the 2 K3s server nodes on that host disappeared simultaneously, dropping K3s's own etcd quorum down to a single surviving node out of three — a majority loss.

2. Why "We Used an Odd Number of Nodes" Wasn't Enough

section02

K3s's embedded etcd is designed around an odd number of server nodes (3 or 5), staying operational as long as a majority remain alive. A guide to configuring K3s with embedded HA (etcd) notes that a 3-node setup tolerates 1 failure, and a 5-node setup tolerates 2, with a recommended latency of under 10ms between server nodes.

That design assumes K3s nodes fail independently. But in practice, since they're VMs, K3s nodes always run on top of some physical host — in this case, Proxmox. K3s's quorum design and Proxmox's quorum design are separate layers, but node placement ties them together.

When 3 K3s nodes are split 2-to-1 across 2 hosts, losing "just one host" — an ordinary-sounding failure — directly becomes "losing 2 K3s nodes at once" on the Kubernetes side, a failure mode the odd-node design never anticipated. Using an odd number of nodes was correct in itself; the placement design simply created a different weak point.

With a managed K3s platform like Kubo, this kind of node placement design spanning both the hypervisor layer and the Kubernetes layer is handled as standard, without operators needing to track it themselves.

3. The Real Culprit: Corosync's Self-Fencing — A Structural Weakness of 2-Node Proxmox Clusters

section03

Proxmox's cluster management relies on Corosync's majority-vote quorum. Proxmox VE's official documentation states that cluster-wide locking requires quorum to be formed, and that while quorum is lost, a node cannot reset its watchdog — once the watchdog timer expires, the entire host reboots after 60 seconds (see Proxmox VE High Availability).

This becomes a structural weakness in 2-node configurations. With 3 nodes, a single failure still leaves a majority (2 of 3 votes). With 2 nodes, the moment one becomes unreachable, the remaining node holds only 1 of 2 votes and loses quorum. Multiple reports on Proxmox's community forums describe one node rebooting and the other also rebooting roughly 60 seconds later due to watchdog expiration, and cases where a broad network interruption — such as a switch restart — caused both nodes to self-fence simultaneously (see Proxmox community forum).

There's an irony here: a safety mechanism built to prevent split-brain can, depending on node placement, become the very trigger that takes down the entire cluster at once.

4. Recovery and Immediate Mitigation — QDevice and cluster-reset

section04

The structural fix on the Proxmox side is introducing a QDevice (corosync-qdevice / corosync-qnetd). By placing an independent external host outside the 2-node cluster and giving it a third vote, the remaining node can still hold 2 of 2 votes if one host goes down, preserving quorum. This is the standard recommended configuration for even-numbered clusters, and especially for 2-node clusters.

If K3s's own etcd quorum does end up broken, k3s server --cluster-reset is the entry point for recovery. According to K3s's official server CLI reference, this flag makes the node "forget all peers and become sole member of a new cluster" — a non-destructive reset. There's a catch, though: if the IP used during restoration doesn't match the original --node-ip setting, that network mismatch can itself become a new source of instability, as documented in a similar report on K3s's official GitHub repository. A real-world example posted on the SUSE Forums confirms that the error "this server is not a member of the etcd cluster" stems from exactly this kind of IP mismatch. It's safer to fix --node-ip permanently in config.yaml rather than specifying it ad hoc on the command line each time.

With Kubo On-Premise, recovery procedures for this kind of quorum loss are built into standard operations, so you're not left cross-referencing runbooks and commands in the middle of an incident.

5. The Real Fix — Treat Node Placement and Clustering Design as Separate Problems

section05

The real lesson from this incident isn't just "2-node clusters have a known weakness," as Proxmox's own forums point out — it's that the host-layer quorum design and the Kubernetes-layer quorum design were never examined as two independent problems. There are three concrete things to get right to prevent a repeat:

  • Spread nodes across hosts: distribute K3s server nodes one-per-host across at least 3 Proxmox hosts. Don't concentrate them on just 2
  • Use a fixed registration address: put a VIP or load balancer in front of the server node group so the cluster doesn't depend on any single node's IP (see the K3s embedded HA configuration guide)
  • Understand the timeout design: etcd's defaults are a 100ms heartbeat interval and a 1000ms election timeout. etcd's official tuning documentation recommends setting the election timeout to at least 10 times the round-trip time between members. If network latency eats into that margin, unnecessary leader elections can fire, destabilizing the cluster temporarily

Both Proxmox's quorum design and K3s's etcd quorum design can be assembled correctly on their own, strictly by following each project's official documentation. The problem is that neither document, on its own, teaches you about the intersection — node placement — where both designs have to hold at the same time.

Summary

Even with an odd number of nodes securing etcd quorum, if you haven't also designed the number and placement of the underlying Proxmox hosts, a hypervisor-layer failure can translate directly into a Kubernetes-layer failure. Proxmox's self-fencing exists to prevent split-brain, but in a 2-node setup with the wrong node placement, that same safety mechanism can become the trigger for the outage it was meant to prevent.

Running Proxmox and K3s yourself means reconciling host count, whether you need a QDevice, K3s node placement, and timeout design, all on your own. Kubo is a managed Kubernetes platform built on K3s with this kind of availability design already built in, and the same design carries over to on-premises environments through Kubo On-Premise. If you'd like to reduce the complexity of running this kind of setup yourself, you can also reach out via our contact page to discuss your architecture.

Related articles

← Back to all posts