Skip to main content

Your RTO Said 30 Seconds. It Took 3 Minutes. Why K3s and kubeadm HA Specs Don't Match Reality

1. K3s and kubeadm Start From Different HA Assumptions

section01

When teams weigh "k3s vs k8s" for production, one thing that gets overlooked is how different the underlying HA (high availability) assumptions really are. K3s and kubeadm (standard Kubernetes) both talk about a "3-node setup," but what that actually means differs significantly.

K3s's HA configuration is built around embedded etcd. According to the K3s official documentation, an HA embedded etcd cluster must be composed of an odd number of server nodes to maintain quorum, with a minimum of 3 nodes. Each server node runs the API server and an etcd member together, keeping the footprint compact.

kubeadm, on the other hand, offers two topologies, as described in the Kubernetes official documentation: stacked etcd and external etcd. Stacked etcd runs etcd as a static pod on each control-plane node and can be built with a minimum of 3 nodes. External etcd separates etcd onto its own set of machines — 3 control-plane nodes plus at least 3 additional etcd nodes, for a minimum of 6 machines total.

In other words, "building an HA cluster" means something very different in terms of infrastructure footprint depending on whether you're running K3s (3 nodes) or kubeadm (3 to 6 nodes, depending on topology).

2. The Gap Between Catalog RTO and Real-World Recovery

section02

When evaluating HA setups, most documentation cites RTO (Recovery Time Objective) targets like "leader election in under 30 seconds" or "traffic failover in under 60 seconds" after a node failure. Operational guidance on K3s HA echoes this: an acceptable RTO for etcd leader election is under 30 seconds, and single-node failure traffic rerouting is commonly expected within 60 seconds.

But these numbers assume ideal conditions. In practice, one real-world test on a common 4-node setup (3 masters + 1 worker) found that failover took 3 to 6 minutes to complete after a single node failure — nearly six times longer than the catalog figure.

This gap comes from several compounding factors beyond etcd's own leader election time: load balancer health-check intervals, DNS/service-discovery propagation delay, and kubelet's node-status detection timeouts all add up. If you design your SLA around a bare "RTO: 30 seconds" figure, you may end up accountable for an outage that runs far longer than promised.

3. Coupled vs. Independent Failure Domains: Stacked vs. External etcd

section03

Beyond how long failover actually takes, there's another critical design axis: what else breaks when one thing breaks — the failure domain design.

With kubeadm's stacked etcd setup, as the comparison of stacked vs. external etcd topologies points out, losing a control-plane node also means losing an etcd member on the same machine. In a 3-node cluster, losing one node is within tolerance, but losing a second immediately breaks quorum — this isn't just reduced capacity, it's a hard stop. General guidance on highly available Kubernetes design similarly recommends spreading master nodes across separate failure domains (availability zones or racks), something a stacked topology effectively collapses onto a single node.

With external etcd, control-plane and etcd failure domains are fully decoupled. A failure in one layer doesn't take down the other, which matters more as cluster size grows. The tradeoff is that you need twice the machine count (3 nodes become 6) and twice the operational surface.

K3s's embedded etcd is architecturally closer to stacked etcd — it trades failure-domain isolation for simplicity. That's a reasonable choice when you want to get a small cluster into production quickly, but how far it scales for large, mission-critical environments depends heavily on your cluster size.

4. The etcd Quorum Formula — and the Odd-Node Trap

section04

Whether you run K3s or kubeadm, fault tolerance ultimately comes down to etcd's quorum. The etcd official documentation explains that under the Raft consensus algorithm, losing a majority of members stops the cluster from accepting writes. The exact majority requirement, as the K3s official documentation spells out, follows the formula (N/2)+1 for an N-node cluster.

Concretely:

  • 3 nodes: quorum is 2, tolerates 1 failure
  • 5 nodes: quorum is 3, tolerates 2 failures
  • 7 nodes: quorum is 4, tolerates 3 failures

One easy-to-miss trap: adding a single node to an odd-sized cluster doesn't necessarily improve fault tolerance. Going from 3 to 4 nodes, for example, keeps quorum at 3 — the tolerated-failure count doesn't change, and you've simply increased the number of members required for agreement. If you want to raise fault tolerance by one level, you need to go to 5 nodes, not add a single even-numbered node.

If a majority of members are lost simultaneously, the etcd cluster stops accepting new writes entirely, and the Kubernetes API server becomes read-only at best, or fully unresponsive at worst. Once a majority comes back online, etcd automatically elects a new leader and recovers; if it doesn't, manual disaster recovery is required. Quorum design should be worked backward from the question: "how many nodes can we lose at once before this affects the business?"

5. Conclusion — Which Should You Run in Production?

section05

Putting this all together, the decision mostly comes down to two factors: cluster scale and how much operational capacity you have.

As noted in a 2026 comparison of K3s and kubeadm, K3s runs on as little as 512MB of RAM per server with a control-plane footprint of roughly 300MB, and can stand up a 3-node HA cluster with just a few commands. For small-to-mid-size production clusters in the 3-to-20-node range, that low build-and-operate cost often outweighs the coupled failure domain tradeoff.

On the other hand, as a breakdown of kubeadm vs. K3s vs. EKS explains, kubeadm shines when you need full control over every component, or when you're running clusters beyond 100 nodes. External etcd lets you isolate failure domains and, if needed, dedicate NVMe storage to etcd to avoid I/O contention — the kind of deep tuning large environments require.

A practical rule of thumb: "start small with K3s embedded etcd if you want to minimize operational load at a 3-node scale," versus "choose kubeadm's external etcd if you expect to scale toward 1,000+ nodes and already have the operations team to support it." Our earlier benchmark comparing K3s and kubeadm memory usage also found K3s ahead on resource efficiency — consistent with the small-to-mid-scale advantage this HA analysis points to.

That said, designing HA topology, monitoring etcd quorum health, and running failover tests yourself takes real expertise and time. Kubo is a managed Kubernetes service built on K3s that bakes in quorum management and failover monitoring by default, giving you the full capabilities of Kubernetes at a lower cost than EKS or AKS. If you want to cut down the engineering time spent designing and validating HA setups, check the pricing page to compare costs against other managed Kubernetes options.

Summary

K3s's embedded etcd and kubeadm's stacked/external etcd may look similar on paper when you compare RTO numbers, but real-world recovery time and failure-domain design differ in meaningful ways.

  • Catalog RTO figures (30-60 seconds) can diverge sharply from real-world measurements (3-6 minutes)
  • Stacked/embedded etcd couples failure domains; external etcd decouples them at the cost of doubling node count
  • Quorum follows (N/2)+1, and adding a single node to an odd cluster doesn't always improve fault tolerance

HA design shouldn't be decided from official RTO targets alone — work backward from your own cluster scale and operational capacity to size nodes and failure domains correctly. If your team wants to reduce the engineering effort spent designing, testing, and monitoring HA setups, Kubo's contact page offers configuration guidance tailored to your cluster scale.

Related articles

← Back to all posts