Skip to main content

Is Longhorn Enough, or Do You Need OpenEBS Mayastor? Deciding K3s Storage by Performance Requirements

When choosing persistent storage for a K3s cluster, many teams default to Longhorn without much thought. It integrates well with Rancher and K3s, has a friendly web UI, and installs with a single Helm chart. That's not a bad choice on its own.

But once you start running IOPS-sensitive workloads in production — saving AI training checkpoints, or writing high-frequency time-series data — you can hit a wall where everything suddenly feels too slow. That's when OpenEBS Mayastor, an engine purpose-built for NVMe (a high-speed flash storage connection standard), enters the conversation.

This article compares Longhorn and OpenEBS Mayastor across four axes — architecture, hardware requirements, replica configuration, and performance characteristics — and provides a checklist for choosing K3s storage without regret. By the way, there's also an option that takes this whole decision off your plate: a managed K3s service like Kubo (more on that later).

Architectural Differences — Longhorn's Replica-Based Design vs. Mayastor's SPDK/NVMe-oF Approach

Longhorn is a distributed block storage system installed as a Kubernetes application. It creates a dedicated controller (frontend) for each volume and synchronously replicates data to replicas placed across multiple nodes. Designed as enterprise-grade distributed storage with no single point of failure, Longhorn's adoption has grown from 2,700 to over 34,000 nodes since it was promoted to a CNCF (Cloud Native Computing Foundation) Incubating project in 2021 (CNCF official blog).

OpenEBS, on the other hand, takes a modular approach that separates the data plane (the layer that actually handles I/O) from the control plane. Today, Mayastor is the engine responsible for "Replicated Volumes" across multiple nodes. The older Jiva and cStor engines have already been deprecated and archived, with Mayastor now positioned as the standard replicated storage engine (OpenEBS official GitHub releases).

At the core of Mayastor is a data plane written in Rust. The control plane handles disk pool management and CSI-compliant provisioning, while on the data plane side, a volume target called "Nexus" bundles multiple storage resources and exposes them as a single volume (OpenEBS official repository). Leveraging SPDK (Storage Performance Development Kit) and NVMe-oF (NVMe over Fabrics) with low-latency I/O via io_uring is the biggest design difference from Longhorn.

section01

The Hardware Fork in the Road — NVMe and HugePages Availability Decide the Answer

Before even talking about performance, the first thing to check when comparing these two is whether you actually have the hardware to run them at all.

Worker nodes running Mayastor need x86-64 CPUs (with SSE4.2 instruction set support) and CPU cores that can be dedicated to the I/O engine. On the memory side, you need at least 1GiB of RAM for the I/O engine, plus at least 2GiB of 2MiB-sized HugePages (a large-page memory management mechanism) — that is, 1,024 pages reserved. NVMe devices and a loaded nvme-tcp kernel module are also prerequisites (OpenEBS official documentation).

In other words, Mayastor simply isn't an option on generic cloud instances or on-premises nodes equipped only with regular SATA/SAS SSDs. Longhorn, by contrast, runs from a single node and requires no special CPU instruction sets or HugePages reservation — just a dedicated disk.

Checkpoint: If your node configuration lacks the NVMe devices and HugePages headroom, Longhorn is the only choice at this point. Forcing Mayastor without meeting its prerequisites means you can't even deploy it.

section02

Comparing Replica Configuration and Data Protection — numberOfReplicas and repl, Snapshots vs. Backups

Longhorn lets you specify the replica count from 1 to 20 via the numberOfReplicas parameter on a StorageClass. The default is 3, but production environments are recommended to reduce it to 2. With 3 replicas, a 100GB volume consumes 300GB of disk, and writes are amplified 3x over the network — so 2 is the realistic sweet spot for balancing availability and efficiency. Data placement is also controlled via the dataLocality parameter, and best-effort (keep one replica on the local node if possible, fall back to remote nodes otherwise) is the recommended default (Longhorn official documentation).

OpenEBS's Replicated PV Mayastor also lets you specify the replica count via the repl parameter, with the control plane managing replica placement, health monitoring, and automatic rebuilds on failure. Note that local-volume engines such as LVM LocalPV and ZFS LocalPV don't support replication at all — pods are scheduled to the node where the volume was provisioned, by design.

Data protection philosophy differs too. Since v1.3, Longhorn has clearly separated CSI snapshot requests into "local snapshots" (stored within each replica) and "remote backups" (stored to NFS or S3-compatible object stores). You can distinguish between the two by specifying type: snap or type: bak in a VolumeSnapshotClass (SUSE official blog). The underlying principle — that replication protects against node failure but is not a substitute for backup — holds true whether you choose Longhorn or Mayastor. Scheduling at least one recurring backup per volume is explicitly listed as a production best practice in Longhorn's official documentation (Longhorn official documentation).

section03

Performance Characteristics — Where the Benchmarks Flip the Ranking

Asking "which one is faster" in general terms isn't really a meaningful question, because the answer flips depending on workload characteristics.

An academic analysis of Longhorn's performance reports a baseline (pre-tuning) configuration achieving roughly 17,000 read IOPS and 13,000 write IOPS, with bandwidth around 300MB/s for both reads and writes. The analysis identifies the controller-to-replica communication layer and the block storage layer as common bottlenecks, suggesting that redesigning these layers can yield a 3-6x improvement in read/write throughput (Academic paper: Longhorn performance optimization analysis). In other words, Longhorn's out-of-the-box IOPS is sufficient for general-purpose workloads, but without tuning, it can feel underwhelming for extremely IOPS-sensitive use cases like high-frequency trading or time-series databases.

Mayastor, by contrast, is designed so that kernel-bypass I/O via SPDK and NVMe-oF can pass through NVMe's native performance to the volume almost unchanged. As long as the prerequisites (NVMe, HugePages) are met, it tends to show a clear advantage for high-IOPS database workloads.

Checkpoint: For typical PostgreSQL or MySQL workloads, Longhorn's performance is often sufficient in practice unless you're dealing with high-concurrency OLTP. If a hard IOPS ceiling is clearly the bottleneck, it's worth considering a switch to Mayastor.

section04

A Decision Framework — Four Checklist Items to Choose Between Them

Here's everything above distilled into four checklist items to confirm before production rollout.

  1. Hardware requirements: Do the target nodes have NVMe devices and at least 2GiB of HugePages headroom? If not, Longhorn is the only option.
  2. Required storage type: Is block storage alone sufficient, or do you need file/object storage unified on the same platform too? Both solutions here are block-oriented; if requirements broaden, an option like Rook/Ceph comes into view.
  3. Workload IOPS profile: Longhorn is sufficient for typical web apps and small-to-mid-size databases. If the IOPS ceiling becomes a real problem — high-frequency trading, time-series databases, AI checkpoint storage — consider Mayastor.
  4. Team familiarity and operational workflow: If you prioritize Rancher/K3s integration, a simple web UI, and easy S3 backup workflows, choose Longhorn. If you want to mix different storage engines per workload and your team is comfortable operating NVMe-dedicated infrastructure, choose Mayastor.

If even one of these four points leans toward "Longhorn," there's no need to force your way toward Mayastor. OpenEBS itself has consolidated around Mayastor (OpenEBS official GitHub releases), and this division of labor — "Mayastor if you can meet the requirements, Longhorn if you can't" — is unlikely to change anytime soon.

Running this checklist yourself, and re-evaluating it every time there's an incident or a version upgrade, is an ongoing burden for infrastructure teams. Kubo Cloud is built on K3s with Prometheus and Grafana monitoring and cert-manager integration standard, reducing the operational load from storage selection all the way through monitoring and certificate management.

If you're also weighing an option like Ceph CSI, we've covered the operational pitfalls you're likely to hit once you're running it in production in the RBD lock trap you should know about before node maintenance on K3s + Ceph CSI on Proxmox.

section05

Conclusion

Longhorn and OpenEBS Mayastor aren't a relationship you can settle by asking which one is "better." Longhorn offers simplicity and ecosystem integration, with the flexibility to run from a single node. Mayastor only delivers its true low-latency performance in environments that can meet its NVMe and HugePages prerequisites.

Longhorn itself has its own operational pitfalls, too. For example, depending on the startup order of the storage network after a node reboot, replica rebuilds can silently stall — an operational issue we cover in detail in the problem where Longhorn replica rebuilds stall after a K3s node reboot. Looking beyond selection criteria to the operational design after deployment is the key to a storage choice you won't regret.

That said, carrying the full weight of selection, tuning, and incident response on your own every time is never a light burden for an infrastructure team. Kubo is a managed Kubernetes built on K3s, with Helm chart support and Prometheus/Grafana monitoring built in standard, taking the time your operations team spends agonizing over storage selection and tuning off your hands. If you need to take advantage of NVMe for a high-performance configuration, or need to keep everything under your own on-premises management, Kubo On-Premise is also an option for a hardware-optimized setup. Start with a contact us to talk through the storage configuration that fits your workload.

Related articles

← Back to all posts