One Morning, a K3s Node Died — and That's When We Found Out
A team running a production K3s cluster had been operating daily Velero backups for over six months. Every night, the notification in Slack read "Completed." Nobody ever questioned whether the backups were actually working.
Then one morning, a worker node failed to boot after a disk failure, taking down the PostgreSQL Pod that had been running on it. The team restored from Velero without hesitation — the Pod and PVC were both created successfully. But when they checked the table contents, it was empty. Not a single row of data had come back.
If you're running Velero on K3s for backup and restore, this isn't a hypothetical. In this article, we break down why a Velero backup on K3s can "succeed" and still fail to restore, how to diagnose it, and what to put in place to make sure it never happens in production.
Why the Data Was Gone Even Though It Said "Completed" — A K3s Storage Assumption
The root cause was the storage provisioner K3s ships with by default.
K3s bundles Rancher's Local Path Provisioner out of the box. When you create a PersistentVolumeClaim, it simply carves out a directory under /var/lib/rancher/k3s/storage on whichever node the Pod happens to be scheduled on (K3s official documentation). The Local Path Provisioner GitHub repository is explicit that this isn't designed for distributed applications that need volume migration, replication, or snapshots — it's meant for cases where persisting node-specific data is good enough, such as development environments or single-node setups.
In other words, Local Path Provisioner doesn't implement the CSI driver capability for snapshots (CREATE_DELETE_SNAPSHOT). When Velero tries to take a CSI snapshot, there's simply no counterpart on the storage side to create one.
And yet the Velero backup job keeps reporting "Completed." That's because Velero itself successfully backs up the Kubernetes resource definitions (the YAML manifests), so the job genuinely does finish without error. Whether the actual volume data was protected is a completely separate question — one that's easy to overlook because nothing in the job status calls it out.
If this assumption had been caught earlier, choosing a managed K3s platform like Kubo, which ships with CSI-compatible storage such as Longhorn by default, would have been an option worth considering.
Pinpointing the Silent Failure Modes in CSI Snapshots
Even setting aside the Local Path Provisioner specifics, the CSI snapshot mechanism itself has several failure points that are easy to miss.
First, the basics: VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass are CRDs in the snapshot.storage.k8s.io group — they are not part of the Kubernetes core API. Installing these CRDs and the snapshot-controller is explicitly the responsibility of the Kubernetes distribution (Kubernetes official documentation). On lightweight distributions like K3s, it's not unusual for this piece to simply be missing.
On top of that, here are some of the common failure patterns (Netdata's guide):
- Missing CRDs or snapshot-controller: the VolumeSnapshot never binds to a VolumeSnapshotContent
- VolumeSnapshotClass driver name mismatch: the sidecar goes completely silent, without even emitting an event
- CSI driver doesn't support snapshots: the sidecar's request gets rejected by the driver as unimplemented
- Wrong storageClassName on restore: instead of restoring from the snapshot, an empty volume gets provisioned from scratch
That last one — a wrong storageClassName — is especially dangerous. The PVC and Pod are both created successfully, so nothing looks wrong on the surface. Nobody notices until they actually check the data.
If you're using Velero, there's one more requirement: the relevant VolumeSnapshotClass must be labeled velero.io/csi-volumesnapshot-class: "true" (Velero's official CSI documentation). Miss that label, and snapshot creation fails outright with an error that no matching VolumeSnapshotClass could be found for the provisioner.
Keeping this configuration consistent on your own, every time, is a real burden for a small team. Kubo Cloud ships with CSI-compatible storage and monitoring built in, which removes the risk of exactly this kind of "forgot one label" incident in the first place.
The Right Fallback — File System Backup, and Its Limits
To actually protect volume data on storage that doesn't support snapshots — like Local Path Provisioner — you need Velero's File System Backup (FSB).
FSB works through a node-agent DaemonSet running on every node, which reads Pod volumes via their hostPath mounts and uploads the data to object storage using Kopia (which replaced the older Restic). For volumes that don't support CSI snapshots — EFS, AzureFile, NFS, and local storage like Local Path Provisioner — FSB is the practical option (Velero's official File System Backup documentation).
But FSB has its own limits. Because it reads directly from a live, running file system, it can't capture a single, consistent point in time the way a snapshot does. The official documentation is explicit that FSB is less consistent than the snapshot-based approach. For workloads like databases, where files are constantly being rewritten, a backup taken this way is at best crash-consistent — roughly equivalent to the state you'd get from suddenly cutting the power.
If you need application-consistent backups, you have to use Velero's backup hooks to freeze the file system right before the backup, or trigger the database's own dump command (Velero's official backup hooks documentation). This can be done with a simple fsfreeze command via Pod annotations, but it also means taking on an extra operational task: writing the hook yourself and verifying it actually works.
Don't Trust the "Backup Succeeded" Log — Make Restore Drills Routine
As we've seen, a "Completed" status from Velero only proves that the backup controller finished its workflow from start to finish. Whether that backup can actually restore your application is a separate thing that has to be verified on its own (Velero's official post on CSI integration).
A common misconception here is thinking "we take etcd snapshots on K3s, so our backups are solid." An etcd snapshot protects Kubernetes resource definitions — the equivalent of your Pod and Service YAML — not the application data sitting on your PersistentVolumes. If a database's PV is lost to a node failure, restoring cluster state from an etcd snapshot still brings the database back up empty.
The way to catch this gap before it bites you in production is to run regular "restore drills": actually restore with Velero into an isolated namespace or cluster, and confirm the application starts correctly and the data is readable. Doing this monthly, or at minimum once a quarter, and recording your measured RTO (recovery time objective) and RPO (recovery point objective) each time, lets you find the gap between "we have a backup" and "we can actually recover" before an incident forces you to find out the hard way.
Running this verification cycle manually is a heavy lift for an infrastructure team that's already stretched thin. Kubo gives you a production-grade K3s-based Kubernetes environment with CSI-compatible storage and operational monitoring built in, so you don't have to design your backup strategy from scratch. On the cost side, Kubo Cloud starts at ¥48,000 per month — about 58% of the cost of EKS — freeing up the budget and time you'd otherwise spend validating your backup infrastructure.
Summary
K3s's default Local Path Provisioner is a simple storage option optimized for development and CI use, and it does not support CSI snapshots. Keep using it as-is in production, and you can end up in a state where Velero backup jobs keep reporting "Completed" while the volume data is never actually protected.
There are three key takeaways:
- Don't run Local Path Provisioner as-is in production: consider migrating to CSI-compatible storage such as Longhorn
- When CSI snapshots aren't available, use File System Backup — and understand its limits: assume it's crash-consistent at best, and pair it with backup hooks where application consistency matters
- Judge your backups by whether you can restore them, not whether they ran: schedule regular restore drills and measure your actual RTO/RPO
If you want to reduce the ongoing burden of designing and validating your backup and disaster recovery strategy, Kubo, with CSI-compatible storage built in, or Kubo On-Premise for fully self-managed data, are both worth a look. Start with a contact request to talk through your current backup setup.