[{"data":1,"prerenderedAt":392},["ShallowReactive",2],{"blog-en-k3s-velero-backup-restore-local-path-provisioner-empty-pvc":3,"blog-related-en-k3s-velero-backup-restore-local-path-provisioner-empty-pvc":331,"blog-en-k3s-velero-backup-restore-local-path-provisioner-empty-pvc-alt":319},{"id":4,"title":5,"author":6,"body":7,"date":313,"description":314,"extension":315,"image":316,"locale":317,"meta":318,"navigation":319,"path":320,"seo":321,"stem":322,"tags":323,"__hash__":330},"blog\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc.md","Every Night the Velero Backup Said \"Completed.\" Then We Restored It on K3s and the PVC Was Empty","Kubo Team",{"type":8,"value":9,"toc":303},"minimark",[10,15,19,22,25,29,36,39,61,68,71,80,84,90,93,118,127,156,163,176,184,188,194,197,206,209,222,226,232,240,243,246,253,257,260,263,284],[11,12,14],"h2",{"id":13},"one-morning-a-k3s-node-died-and-thats-when-we-found-out","One Morning, a K3s Node Died — and That's When We Found Out",[16,17,18],"p",{},"A team running a production K3s cluster had been operating daily Velero backups for over six months. Every night, the notification in Slack read \"Completed.\" Nobody ever questioned whether the backups were actually working.",[16,20,21],{},"Then one morning, a worker node failed to boot after a disk failure, taking down the PostgreSQL Pod that had been running on it. The team restored from Velero without hesitation — the Pod and PVC were both created successfully. But when they checked the table contents, it was empty. Not a single row of data had come back.",[16,23,24],{},"If you're running Velero on K3s for backup and restore, this isn't a hypothetical. In this article, we break down why a Velero backup on K3s can \"succeed\" and still fail to restore, how to diagnose it, and what to put in place to make sure it never happens in production.",[11,26,28],{"id":27},"why-the-data-was-gone-even-though-it-said-completed-a-k3s-storage-assumption","Why the Data Was Gone Even Though It Said \"Completed\" — A K3s Storage Assumption",[16,30,31],{},[32,33],"img",{"alt":34,"src":35},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc\u002Fsection01.webp",[16,37,38],{},"The root cause was the storage provisioner K3s ships with by default.",[16,40,41,42,46,47,54,55,60],{},"K3s bundles Rancher's Local Path Provisioner out of the box. When you create a PersistentVolumeClaim, it simply carves out a directory under ",[43,44,45],"code",{},"\u002Fvar\u002Flib\u002Francher\u002Fk3s\u002Fstorage"," on whichever node the Pod happens to be scheduled on (",[48,49,53],"a",{"href":50,"rel":51},"https:\u002F\u002Fdocs.k3s.io\u002Fadd-ons\u002Fstorage",[52],"nofollow","K3s official documentation","). The ",[48,56,59],{"href":57,"rel":58},"https:\u002F\u002Fgithub.com\u002Francher\u002Flocal-path-provisioner",[52],"Local Path Provisioner GitHub repository"," is explicit that this isn't designed for distributed applications that need volume migration, replication, or snapshots — it's meant for cases where persisting node-specific data is good enough, such as development environments or single-node setups.",[16,62,63,64,67],{},"In other words, Local Path Provisioner doesn't implement the CSI driver capability for snapshots (",[43,65,66],{},"CREATE_DELETE_SNAPSHOT","). When Velero tries to take a CSI snapshot, there's simply no counterpart on the storage side to create one.",[16,69,70],{},"And yet the Velero backup job keeps reporting \"Completed.\" That's because Velero itself successfully backs up the Kubernetes resource definitions (the YAML manifests), so the job genuinely does finish without error. Whether the actual volume data was protected is a completely separate question — one that's easy to overlook because nothing in the job status calls it out.",[16,72,73,74,79],{},"If this assumption had been caught earlier, choosing a managed K3s platform like ",[48,75,78],{"href":76,"rel":77},"https:\u002F\u002Fkubo.hexabase.io\u002F",[52],"Kubo",", which ships with CSI-compatible storage such as Longhorn by default, would have been an option worth considering.",[11,81,83],{"id":82},"pinpointing-the-silent-failure-modes-in-csi-snapshots","Pinpointing the Silent Failure Modes in CSI Snapshots",[16,85,86],{},[32,87],{"alt":88,"src":89},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc\u002Fsection02.webp",[16,91,92],{},"Even setting aside the Local Path Provisioner specifics, the CSI snapshot mechanism itself has several failure points that are easy to miss.",[16,94,95,96,99,100,103,104,107,108,111,112,117],{},"First, the basics: ",[43,97,98],{},"VolumeSnapshot",", ",[43,101,102],{},"VolumeSnapshotContent",", and ",[43,105,106],{},"VolumeSnapshotClass"," are CRDs in the ",[43,109,110],{},"snapshot.storage.k8s.io"," group — they are not part of the Kubernetes core API. Installing these CRDs and the snapshot-controller is explicitly the responsibility of the Kubernetes distribution (",[48,113,116],{"href":114,"rel":115},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fvolume-snapshots\u002F",[52],"Kubernetes official documentation","). On lightweight distributions like K3s, it's not unusual for this piece to simply be missing.",[16,119,120,121,126],{},"On top of that, here are some of the common failure patterns (",[48,122,125],{"href":123,"rel":124},"https:\u002F\u002Fwww.netdata.cloud\u002Fguides\u002Fkubernetes\u002Fkubernetes-volume-snapshot-failures\u002F",[52],"Netdata's guide","):",[128,129,130,138,144,150],"ul",{},[131,132,133,137],"li",{},[134,135,136],"strong",{},"Missing CRDs or snapshot-controller",": the VolumeSnapshot never binds to a VolumeSnapshotContent",[131,139,140,143],{},[134,141,142],{},"VolumeSnapshotClass driver name mismatch",": the sidecar goes completely silent, without even emitting an event",[131,145,146,149],{},[134,147,148],{},"CSI driver doesn't support snapshots",": the sidecar's request gets rejected by the driver as unimplemented",[131,151,152,155],{},[134,153,154],{},"Wrong storageClassName on restore",": instead of restoring from the snapshot, an empty volume gets provisioned from scratch",[16,157,158,159,162],{},"That last one — a wrong ",[43,160,161],{},"storageClassName"," — is especially dangerous. The PVC and Pod are both created successfully, so nothing looks wrong on the surface. Nobody notices until they actually check the data.",[16,164,165,166,169,170,175],{},"If you're using Velero, there's one more requirement: the relevant VolumeSnapshotClass must be labeled ",[43,167,168],{},"velero.io\u002Fcsi-volumesnapshot-class: \"true\""," (",[48,171,174],{"href":172,"rel":173},"https:\u002F\u002Fvelero.io\u002Fdocs\u002Fv1.13\u002Fcsi\u002F",[52],"Velero's official CSI documentation","). Miss that label, and snapshot creation fails outright with an error that no matching VolumeSnapshotClass could be found for the provisioner.",[16,177,178,179,183],{},"Keeping this configuration consistent on your own, every time, is a real burden for a small team. ",[48,180,182],{"href":76,"rel":181},[52],"Kubo Cloud"," ships with CSI-compatible storage and monitoring built in, which removes the risk of exactly this kind of \"forgot one label\" incident in the first place.",[11,185,187],{"id":186},"the-right-fallback-file-system-backup-and-its-limits","The Right Fallback — File System Backup, and Its Limits",[16,189,190],{},[32,191],{"alt":192,"src":193},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc\u002Fsection03.webp",[16,195,196],{},"To actually protect volume data on storage that doesn't support snapshots — like Local Path Provisioner — you need Velero's File System Backup (FSB).",[16,198,199,200,205],{},"FSB works through a node-agent DaemonSet running on every node, which reads Pod volumes via their hostPath mounts and uploads the data to object storage using Kopia (which replaced the older Restic). For volumes that don't support CSI snapshots — EFS, AzureFile, NFS, and local storage like Local Path Provisioner — FSB is the practical option (",[48,201,204],{"href":202,"rel":203},"https:\u002F\u002Fvelero.io\u002Fdocs\u002Fv1.13\u002Ffile-system-backup\u002F",[52],"Velero's official File System Backup documentation",").",[16,207,208],{},"But FSB has its own limits. Because it reads directly from a live, running file system, it can't capture a single, consistent point in time the way a snapshot does. The official documentation is explicit that FSB is less consistent than the snapshot-based approach. For workloads like databases, where files are constantly being rewritten, a backup taken this way is at best crash-consistent — roughly equivalent to the state you'd get from suddenly cutting the power.",[16,210,211,212,217,218,221],{},"If you need application-consistent backups, you have to use Velero's backup hooks to freeze the file system right before the backup, or trigger the database's own dump command (",[48,213,216],{"href":214,"rel":215},"https:\u002F\u002Fvelero.io\u002Fdocs\u002Fv1.9\u002Fbackup-hooks\u002F",[52],"Velero's official backup hooks documentation","). This can be done with a simple ",[43,219,220],{},"fsfreeze"," command via Pod annotations, but it also means taking on an extra operational task: writing the hook yourself and verifying it actually works.",[11,223,225],{"id":224},"dont-trust-the-backup-succeeded-log-make-restore-drills-routine","Don't Trust the \"Backup Succeeded\" Log — Make Restore Drills Routine",[16,227,228],{},[32,229],{"alt":230,"src":231},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc\u002Fsection04.webp",[16,233,234,235,205],{},"As we've seen, a \"Completed\" status from Velero only proves that the backup controller finished its workflow from start to finish. Whether that backup can actually restore your application is a separate thing that has to be verified on its own (",[48,236,239],{"href":237,"rel":238},"https:\u002F\u002Fvelero.io\u002Fblog\u002Fcsi-integration",[52],"Velero's official post on CSI integration",[16,241,242],{},"A common misconception here is thinking \"we take etcd snapshots on K3s, so our backups are solid.\" An etcd snapshot protects Kubernetes resource definitions — the equivalent of your Pod and Service YAML — not the application data sitting on your PersistentVolumes. If a database's PV is lost to a node failure, restoring cluster state from an etcd snapshot still brings the database back up empty.",[16,244,245],{},"The way to catch this gap before it bites you in production is to run regular \"restore drills\": actually restore with Velero into an isolated namespace or cluster, and confirm the application starts correctly and the data is readable. Doing this monthly, or at minimum once a quarter, and recording your measured RTO (recovery time objective) and RPO (recovery point objective) each time, lets you find the gap between \"we have a backup\" and \"we can actually recover\" before an incident forces you to find out the hard way.",[16,247,248,249,252],{},"Running this verification cycle manually is a heavy lift for an infrastructure team that's already stretched thin. ",[48,250,78],{"href":76,"rel":251},[52]," gives you a production-grade K3s-based Kubernetes environment with CSI-compatible storage and operational monitoring built in, so you don't have to design your backup strategy from scratch. On the cost side, Kubo Cloud starts at ¥48,000 per month — about 58% of the cost of EKS — freeing up the budget and time you'd otherwise spend validating your backup infrastructure.",[11,254,256],{"id":255},"summary","Summary",[16,258,259],{},"K3s's default Local Path Provisioner is a simple storage option optimized for development and CI use, and it does not support CSI snapshots. Keep using it as-is in production, and you can end up in a state where Velero backup jobs keep reporting \"Completed\" while the volume data is never actually protected.",[16,261,262],{},"There are three key takeaways:",[264,265,266,272,278],"ol",{},[131,267,268,271],{},[134,269,270],{},"Don't run Local Path Provisioner as-is in production",": consider migrating to CSI-compatible storage such as Longhorn",[131,273,274,277],{},[134,275,276],{},"When CSI snapshots aren't available, use File System Backup — and understand its limits",": assume it's crash-consistent at best, and pair it with backup hooks where application consistency matters",[131,279,280,283],{},[134,281,282],{},"Judge your backups by whether you can restore them, not whether they ran",": schedule regular restore drills and measure your actual RTO\u002FRPO",[16,285,286,287,290,291,296,297,302],{},"If you want to reduce the ongoing burden of designing and validating your backup and disaster recovery strategy, ",[48,288,78],{"href":76,"rel":289},[52],", with CSI-compatible storage built in, or ",[48,292,295],{"href":293,"rel":294},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[52],"Kubo On-Premise"," for fully self-managed data, are both worth a look. Start with a ",[48,298,301],{"href":299,"rel":300},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[52],"contact request"," to talk through your current backup setup.",{"title":304,"searchDepth":305,"depth":305,"links":306},"",2,[307,308,309,310,311,312],{"id":13,"depth":305,"text":14},{"id":27,"depth":305,"text":28},{"id":82,"depth":305,"text":83},{"id":186,"depth":305,"text":187},{"id":224,"depth":305,"text":225},{"id":255,"depth":305,"text":256},"2026-10-13","K3s's default local-path-provisioner doesn't support CSI snapshots. Velero backups can keep reporting \"Completed\" while the volume data is never actually protected, leaving restored PVCs empty. Here's how to diagnose it and fix it with File System Backup and restore drills.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc",{"title":5,"description":314},"blog\u002Fen\u002Fk3s-velero-backup-restore-local-path-provisioner-empty-pvc",[324,325,326,327,328,329],"k3s","kubernetes","velero","backup","disaster-recovery","csi","_wNnJNln6tIve8ZP7rCvyE_R8LJucKaz74u9E-I_Ac0",[332,341,350,360,372,382],{"path":333,"title":334,"description":335,"date":336,"tags":337},"\u002Fblog\u002Fen\u002Fk3s-sealed-secrets-gitops-rebuild-private-key-lost","We Had It All as Code in GitOps. Then We Rebuilt the K3s Cluster, and Every Sealed Secret Became Undecryptable","Rebuilding a K3s cluster with GitOps left every Sealed Secret undecryptable. The real cause was how the private key was handled. Here's the failure mechanism, the recovery steps, and how to prevent it from happening again.","2026-10-15",[324,325,338,339,340,328],"sealed-secrets","secrets-management","gitops",{"path":342,"title":343,"description":344,"date":345,"tags":346},"\u002Fblog\u002Fen\u002Fk3s-etcd-snapshot-restore-token-cluster-reset-pitfalls","The Automated etcd Snapshots Were Running Every Day. Here's Why K3s Restore Still Failed in Production","K3s's etcd snapshots looked healthy every day, yet restore failed during a real outage. Three real-world traps — token management, S3 restore incidents, and cluster-reset pitfalls — reveal what's really at risk in embedded etcd backups.","2026-09-21",[324,325,347,348,328,349],"etcd","backup-restore","high-availability",{"path":351,"title":352,"description":353,"date":354,"tags":355},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[324,325,356,357,358,359],"hpa","containerresource","autoscaling","service-mesh",{"path":361,"title":362,"description":363,"date":364,"tags":365},"\u002Fblog\u002Fen\u002Fk3s-vs-kubernetes-cni-flannel-default-comparison","K3s vs Standard Kubernetes: How Different Their CNI Starting Points Really Are","K3s ships with Flannel built in, while standard Kubernetes starts with no CNI selected at all. This comparison, grounded in the official K3s, Kubernetes, Calico, and Cilium docs, explains how that difference affects NetworkPolicy enforcement and network performance.","2026-10-20",[324,325,366,367,368,369,370,371],"cni","flannel","calico","cilium","networkpolicy","networking",{"path":373,"title":374,"description":375,"date":376,"tags":377},"\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending","We Drained a Node and a Pod Never Came Back. Why Topology Spread Constraints and PodDisruptionBudget Collide on K3s","Draining a node in production K3s left a Pod stuck Pending. The cause: Topology Spread Constraints' maxSkew colliding with PodDisruptionBudget. We verify how draining itself creates a node shortage, based on official documentation.","2026-10-19",[324,325,378,379,380,381],"topology-spread-constraints","poddisruptionbudget","scheduling","node-drain",{"path":383,"title":384,"description":385,"date":386,"tags":387},"\u002Fblog\u002Fen\u002Fkyverno-validatingadmissionpolicy-cel-migration-silent-bypass","Is the CEL Rewrite of Your Policy Really the Same? How Migrating from Kyverno to ValidatingAdmissionPolicy Let Some Resources Silently Slip Through","As Kyverno's ClusterPolicy heads toward deprecation, teams migrating to ValidatingAdmissionPolicy (CEL) have found that validation silently skipped for certain resource types. Here's the CEL runtime error and failurePolicy trap behind it, and a safe migration path.","2026-10-18",[324,325,388,389,390,391],"validatingadmissionpolicy","kyverno","cel","admission-control",1792592640542]