[{"data":1,"prerenderedAt":549},["ShallowReactive",2],{"blog-en-docker-compose-k3s-local-path-provisioner-permission-denied":3,"blog-related-en-docker-compose-k3s-local-path-provisioner-permission-denied":489,"blog-en-docker-compose-k3s-local-path-provisioner-permission-denied-alt":479},{"id":4,"title":5,"author":6,"body":7,"date":474,"description":475,"extension":476,"image":22,"locale":477,"meta":478,"navigation":479,"path":480,"seo":481,"stem":482,"tags":483,"__hash__":488},"blog\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied.md","The Volume That Worked Fine in Docker Compose Threw Permission Denied the Moment It Hit K3s","Kubo Team",{"type":8,"value":9,"toc":466},"minimark",[10,15,23,35,46,49,53,59,80,107,124,128,134,157,230,250,254,260,263,320,329,333,339,346,367,389,392,396,399,431,443,462],[11,12,14],"h2",{"id":13},"_1-nobody-thought-about-who-owns-the-volume-in-docker-compose","1. Nobody Thought About \"Who Owns the Volume\" in Docker Compose",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"title","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied\u002Ftitle.webp",[16,24,25,26,30,31,34],{},"When you're doing local development with Docker Compose, few people ever stop to think about who actually owns a volume. Write ",[27,28,29],"code",{},"volumes: .\u002Fdata:\u002Fapp\u002Fdata"," in your ",[27,32,33],{},"docker-compose.yml",", and the host directory just shows up inside the container. Permission issues almost never come up.",[16,36,37,38,45],{},"That's expected behavior: Docker's own documentation explains that bind mounts have write access to files on the host by default (",[39,40,44],"a",{"href":41,"rel":42},"https:\u002F\u002Fdocs.docker.com\u002Fengine\u002Fstorage\u002Fbind-mounts\u002F",[43],"nofollow","Docker Docs: Bind mounts","). In other words, whatever permissions the host directory has get reflected straight into the container — Docker doesn't add a separate permission layer of its own. If you create the directory with permissive enough settings for development, most applications just work without anyone thinking about it.",[16,47,48],{},"That exact assumption — \"it just works\" — is what trips people up first when migrating from Docker Compose to K3s. K3s (and Kubernetes clusters in general) takes a fundamentally different approach to volumes than Compose does. The more you hand-assemble the configuration yourself, the more of these assumption gaps you end up discovering and fixing one by one.",[11,50,52],{"id":51},"_2-what-happens-the-moment-you-bring-it-to-k3s-the-local-path-provisioner-behavior","2. What Happens the Moment You Bring It to K3s — The local-path-provisioner Behavior",[16,54,55],{},[19,56],{"alt":57,"src":58},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied\u002Fsection01.webp",[16,60,61,62,65,66,71,72,75,76,79],{},"K3s ships with Rancher's ",[27,63,64],{},"local-path-provisioner"," built in, so simply creating a PVC (PersistentVolumeClaim) automatically provisions a volume directory on the node's local disk (",[39,67,70],{"href":68,"rel":69},"https:\u002F\u002Fdocs.k3s.io\u002Fadd-ons\u002Fstorage",[43],"K3s Docs: Volumes and Storage","). By default it uses the ",[27,73,74],{},"local-path"," StorageClass, and the actual volume data lives under ",[27,77,78],{},"\u002Fvar\u002Flib\u002Francher\u002Fk3s\u002Fstorage",".",[16,81,82,83,87,88,90,91,94,95,100,101,106],{},"The catch is that this provisioning process ",[84,85,86],"strong",{},"doesn't take care of directory ownership (chown)",". Looking at the setup script in the official ",[27,89,64],{}," repository, the only operation it runs is ",[27,92,93],{},"mkdir -m 0777 -p \"$VOL_DIR\""," — the permission bits are set to 777 (world read\u002Fwrite), but the owning user and group are never changed (",[39,96,99],{"href":97,"rel":98},"https:\u002F\u002Fgithub.com\u002Francher\u002Flocal-path-provisioner",[43],"rancher\u002Flocal-path-provisioner",", ",[39,102,105],{"href":103,"rel":104},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fblob\u002F7f659759dd006ba2e5156198dde9ca8bf8a6d4e1\u002Fmanifests\u002Flocal-storage.yaml",[43],"k3s-io\u002Fk3s local-storage.yaml",").",[16,108,109,110,113,114,117,118,123],{},"You'd think 777 permissions mean anyone can write to it. In practice, many official container images are built around ",[27,111,112],{},"runAsNonRoot"," or a specific UID\u002FGID, and depending on how ",[27,115,116],{},"securityContext.runAsUser"," is set on the Pod — or how the image itself is designed to start as a non-root user — combined with SELinux settings or umask on the node, write failures have been reported in the wild (",[39,119,122],{"href":120,"rel":121},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F10130",[43],"k3s-io\u002Fk3s Issue #10130","). A question that never surfaced with Docker Compose's bind mounts — \"who exactly is writing here?\" — suddenly becomes impossible to ignore on K3s.",[11,125,127],{"id":126},"_3-relative-paths-are-a-trap-too-hostpath-requires-an-absolute-path","3. Relative Paths Are a Trap Too — hostPath Requires an Absolute Path",[16,129,130],{},[19,131],{"alt":132,"src":133},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied\u002Fsection02.webp",[16,135,136,137,140,141,143,144,147,148,151,152,106],{},"If you're used to Docker Compose's bind mounts, there's another easy-to-miss assumption. Compose's ",[27,138,139],{},"volumes:"," accepts paths relative to ",[27,142,33],{},", but a Kubernetes ",[27,145,146],{},"hostPath"," volume ",[84,149,150],{},"won't work unless it's an absolute path"," (",[39,153,156],{"href":154,"rel":155},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fvolumes\u002F#hostpath",[43],"Kubernetes Docs: Volumes",[158,159,164],"pre",{"className":160,"code":161,"language":162,"meta":163,"style":163},"language-yaml shiki shiki-themes tokyo-night","volumes:\n  - name: data-volume\n    hostPath:\n      path: \u002Fmnt\u002Fdata  # must be absolute; relative paths don't work\n      type: Directory\n","yaml","",[27,165,166,179,196,204,219],{"__ignoreMap":163},[167,168,171,175],"span",{"class":169,"line":170},"line",1,[167,172,174],{"class":173},"s0U2E","volumes",[167,176,178],{"class":177},"sAklC",":\n",[167,180,182,186,189,192],{"class":169,"line":181},2,[167,183,185],{"class":184},"sgJMe","  -",[167,187,188],{"class":173}," name",[167,190,191],{"class":177},":",[167,193,195],{"class":194},"sPY7s"," data-volume\n",[167,197,199,202],{"class":169,"line":198},3,[167,200,201],{"class":173},"    hostPath",[167,203,178],{"class":177},[167,205,207,210,212,215],{"class":169,"line":206},4,[167,208,209],{"class":173},"      path",[167,211,191],{"class":177},[167,213,214],{"class":194}," \u002Fmnt\u002Fdata",[167,216,218],{"class":217},"sbD-w","  # must be absolute; relative paths don't work\n",[167,220,222,225,227],{"class":169,"line":221},5,[167,223,224],{"class":173},"      type",[167,226,191],{"class":177},[167,228,229],{"class":194}," Directory\n",[16,231,232,233,236,237,239,240,243,244,246,247,249],{},"Carry over a Compose-era habit of writing a relative path like ",[27,234,235],{},".\u002Fdata"," into your manifest, and Kubernetes simply can't locate that directory anywhere on the host — the Pod gets stuck with something like CreateContainerConfigError. On top of that, ",[27,238,146],{}," grants direct access to the node's filesystem, which is a fairly powerful privilege, so the Kubernetes documentation itself recommends limiting it to the minimum necessary directories in production and adding ",[27,241,242],{},"readOnly: true"," where possible. In practice, it's safer to lean on the PVC-backed volumes that ",[27,245,64],{}," generates automatically, and keep direct ",[27,248,146],{}," usage to a minimum.",[11,251,253],{"id":252},"_4-the-fix-fsgroup-initcontainers-and-custom-storageclass-patterns","4. The Fix — fsGroup, initContainers, and Custom StorageClass Patterns",[16,255,256],{},[19,257],{"alt":258,"src":259},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied\u002Fsection03.webp",[16,261,262],{},"The good news is that there are a few built-in mechanisms for solving this permission problem without writing anything elaborate yourself.",[264,265,266,290,314],"ul",{},[267,268,269,275,276,279,280,285,286,289],"li",{},[84,270,271,272],{},"Set ",[27,273,274],{},"securityContext.fsGroup",": Specify ",[27,277,278],{},"fsGroup"," in the Pod's securityContext, and Kubernetes automatically changes the volume's group ownership to that GID before the container starts (",[39,281,284],{"href":282,"rel":283},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fconfigure-pod-container\u002Fsecurity-context\u002F",[43],"Kubernetes Docs: Security Context","). Combine it with ",[27,287,288],{},"runAsUser"," and even a non-root container can write to the volume. This is the simplest option to try first when setting things up.",[267,291,292,306,307,309,310,313],{},[84,293,294,295,298,299,298,302,305],{},"Customize ",[27,296,297],{},"userPattern","\u002F",[27,300,301],{},"groupPattern",[27,303,304],{},"permPattern"," on the StorageClass",": ",[27,308,64],{}," supports StorageClass parameters that let you change the ownership and permission pattern used when it creates directories (",[39,311,99],{"href":97,"rel":312},[43],"). This is a good fit if your whole team standardizes on one UID\u002FGID.",[267,315,316,319],{},[84,317,318],{},"chown it in an initContainer",": For the edge cases where fsGroup doesn't take effect (certain CSI drivers, legacy images), a root-privileged initContainer that explicitly chowns the target directory before the main container starts is still a reliable fallback.",[16,321,322,323,328],{},"Whichever approach you pick, the underlying point doesn't change: the volume ownership that Docker Compose never made you think about has to be explicitly designed on K3s. Deciding between fsGroup and a custom StorageClass pattern is exactly the kind of judgment call that eats up time when you haven't hit it before. With ",[39,324,327],{"href":325,"rel":326},"https:\u002F\u002Fkubo.hexabase.io\u002F",[43],"Kubo","'s AI-Driven Deployment, you can describe your application's requirements and get a proposed volume permission design up front, cutting down on the trial-and-error.",[11,330,332],{"id":331},"_5-depends_on-disappears-too-compose-configs-built-around-startup-order-dont-just-work-as-is","5. depends_on Disappears Too — Compose Configs Built Around Startup Order Don't Just Work as-is",[16,334,335],{},[19,336],{"alt":337,"src":338},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied\u002Fsection04.webp",[16,340,341,342,345],{},"Alongside volume permissions, startup order is another thing that's easy to overlook during migration. Docker Compose's ",[27,343,344],{},"depends_on"," is a convenient feature that guarantees container startup order, but Kubernetes has no equivalent concept — Pods generally start in parallel.",[16,347,348,349,352,353,355,356,100,361,366],{},"Even when you mechanically convert a Compose definition into Kubernetes manifests using ",[27,350,351],{},"kompose",", the ",[27,354,344],{}," key is simply ignored (",[39,357,360],{"href":358,"rel":359},"https:\u002F\u002Fgithub.com\u002Fjvitor83\u002Fkompose-network\u002Fblob\u002F16c4818f90e7\u002Fdocs\u002Fconversion.md",[43],"kompose official conversion doc",[39,362,365],{"href":363,"rel":364},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-02-08-how-to-convert-docker-compose-files-to-kubernetes-manifests-with-kompose\u002Fview",[43],"OneUptime: Converting with Kompose","). It's common for the application container to start before the database container, crash-loop on a connection error, and present itself as the next wall right after you've just solved the volume permission problem.",[16,368,369,370,373,374,377,378,383,384,106],{},"The standard fix is to add logic in an ",[27,371,372],{},"initContainers"," step that waits until a dependency like the database is reachable (something like ",[27,375,376],{},"until nslookup db-service; do sleep 2; done","), or to implement connection retries in the application itself (",[39,379,382],{"href":380,"rel":381},"https:\u002F\u002Fbesthub.dev\u002Farticles\u002F10-common-pitfalls-when-migrating-docker-compose-to-kubernetes-a9f6b6310605",[43],"10 Common Pitfalls When Migrating Docker Compose to Kubernetes","). Name resolution between services, on the other hand, is conceptually close between Docker Compose's built-in DNS and Kubernetes' CoreDNS, so that part tends to be a relatively easy piece of the migration (",[39,385,388],{"href":386,"rel":387},"https:\u002F\u002Fkindatechnical.com\u002Fnetworking\u002Fservice-discovery-dns-containers.html",[43],"Service Discovery and DNS in Containers",[16,390,391],{},"This pattern — something that worked as an implicit assumption locally now demanding an explicit design decision on K3s — shows up in both volume permissions and startup order.",[11,393,395],{"id":394},"summary","Summary",[16,397,398],{},"Boiling down the common stumbling blocks when migrating from Docker Compose to K3s gives you three points:",[400,401,402,413,422],"ol",{},[267,403,404,306,407,409,410,412],{},[84,405,406],{},"Volume permissions",[27,408,64],{}," only creates the directory with mode 777 — it never chowns it. You need to explicitly design ownership using ",[27,411,278],{}," or a custom StorageClass pattern.",[267,414,415,418,419,421],{},[84,416,417],{},"Path format",": Compose's relative paths don't work with K3s's ",[27,420,146],{},". Absolute paths are required.",[267,423,424,427,428,430],{},[84,425,426],{},"Startup order",": There's no Kubernetes equivalent to ",[27,429,344],{}," — you need to substitute it with an initContainer or application-level retries.",[16,432,433,434,437,438,442],{},"Each of these is a trap built on the assumption that \"if it worked locally, it'll work in production too.\" These kinds of early setup pitfalls keep recurring for as long as you're hand-assembling the configuration from scratch. With ",[39,435,327],{"href":325,"rel":436},[43],"'s AI-Driven Deployment, you can work through volume permission design and StorageClass selection using natural-language instructions, cutting down on the time spent writing YAML line by line and tracking down the cause of a permission error. The full picture of your setup stays visible in ",[39,439,441],{"href":325,"rel":440},[43],"Kubo Captain UI",", so you're never left guessing at a black box.",[16,444,445,446,451,452,455,456,461],{},"If you want fewer do-overs migrating from Docker Compose — or you're stuck on a Permission denied error right now — you can also reach out via ",[39,447,450],{"href":448,"rel":449},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[43],"Contact Us"," for a configuration consultation. ",[39,453,327],{"href":325,"rel":454},[43],", built on K3s, aims to give you the power of EKS\u002FAKS with the ease of Vercel, and it's designed with exactly these kinds of early migration pitfalls in mind. If there's a reason you can't send data outside your own environment, ",[39,457,460],{"href":458,"rel":459},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[43],"Kubo On-Premise"," offers the same K3s-based foundation built entirely within your own infrastructure.",[463,464,465],"style",{},"html pre.shiki code .s0U2E, html code.shiki .s0U2E{--shiki-default:#F7768E}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sgJMe, html code.shiki .sgJMe{--shiki-default:#9ABDF5}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html pre.shiki code .sbD-w, html code.shiki .sbD-w{--shiki-default:#51597D;--shiki-default-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":163,"searchDepth":181,"depth":181,"links":467},[468,469,470,471,472,473],{"id":13,"depth":181,"text":14},{"id":51,"depth":181,"text":52},{"id":126,"depth":181,"text":127},{"id":252,"depth":181,"text":253},{"id":331,"depth":181,"text":332},{"id":394,"depth":181,"text":395},"2026-11-05","Migrating from Docker Compose to K3s? A volume that worked fine locally suddenly throws Permission denied. The cause is local-path-provisioner's design. Here's how to work around it.","md","en",{},true,"\u002Fblog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied",{"title":5,"description":475},"blog\u002Fen\u002Fdocker-compose-k3s-local-path-provisioner-permission-denied",[484,485,486,64,487],"k3s","kubernetes","docker-compose","persistent-storage","UZBoneYQtlFzCzXRgmPw4-oD94t9CPWAvqjQYEQlfx0",[490,499,509,519,529,539],{"path":491,"title":492,"description":493,"date":494,"tags":495},"\u002Fblog\u002Fen\u002Fk3s-longhorn-storage-network-multus-replica-rebuild-stall","We Rebooted a K3s Node and Longhorn's Replica Rebuild Silently Stalled at 0%. The Storage Network Startup Race Nobody Warned Us About","A K3s node reboot left a Longhorn replica rebuild stuck at 0%. Based on a real bug report, we break down the Storage Network (Multus) startup race, replica count design, and node drain pitfalls for running Longhorn safely in production.","2026-09-23",[484,485,496,497,498,487],"longhorn","storage-network","multus",{"path":500,"title":501,"description":502,"date":503,"tags":504},"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","2026-09-12",[484,485,505,64,506,507,508],"storage","persistent-volume","node-affinity","zfs",{"path":510,"title":511,"description":512,"date":513,"tags":514},"\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap","We Couldn't Reconstruct What an AI Agent Did to Our Kubernetes API. The Gap Was in Our Audit Log Settings","After running AI agents on Kubernetes, we realized that when something went wrong, the Audit Log alone couldn't tell us who did what, or why. Here's the Audit Policy blind spot and how we fixed agent traceability.","2026-11-09",[484,485,515,516,517,518],"ai-agent","audit-log","rbac","observability",{"path":520,"title":521,"description":522,"date":523,"tags":524},"\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure","K3s Pods Kept Getting Evicted. Disk Space Was Fine — Inodes Were Not","K3s pods kept getting evicted while df -h looked perfectly healthy. The real cause wasn't disk space but inode exhaustion. We cover kubelet's eviction thresholds, how to diagnose with df -i, and how to prevent it.","2026-11-08",[484,485,525,526,527,528],"kubelet","eviction","diskpressure","troubleshooting",{"path":530,"title":531,"description":532,"date":533,"tags":534},"\u002Fblog\u002Fen\u002Fk3s-cilium-kube-proxy-replacement-nodeport-unreachable","We Swapped kube-proxy for Cilium on K3s, and NodePort Stopped Working","We followed the official Cilium guide to enable kube-proxy replacement on K3s, and NodePort traffic suddenly stopped reaching our pods. The cause: K3s-specific defaults (Flannel, ServiceLB) that the generic docs never mention. Here's the fix and a pre-production checklist.","2026-11-07",[484,485,535,536,537,538],"cilium","ebpf","kube-proxy","networking",{"path":540,"title":541,"description":542,"date":543,"tags":544},"\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling","requests and limits were correct, but my K3s Pod got slow anyway. The culprit wasn't OOMKill — it was cgroup v2's memory.high throttling early","A K3s Pod with correctly configured requests\u002Flimits suddenly got slow, with no OOMKilled event anywhere. The real cause was cgroup v2's memory.high throttling. Here's how to diagnose and fix it.","2026-11-06",[484,485,545,546,547,548],"cgroup-v2","memory-qos","memory-high","resource-management",1794305502492]