[{"data":1,"prerenderedAt":413},["ShallowReactive",2],{"blog-en-k3s-spegel-embedded-registry-mirror-airgap-node-join":3,"blog-related-en-k3s-spegel-embedded-registry-mirror-airgap-node-join":362,"blog-en-k3s-spegel-embedded-registry-mirror-airgap-node-join-alt":350},{"id":4,"title":5,"author":6,"body":7,"date":344,"description":345,"extension":346,"image":347,"locale":348,"meta":349,"navigation":350,"path":351,"seo":352,"stem":353,"tags":354,"__hash__":361},"blog\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join.md","K3s's Embedded Registry Mirror Was Supposed to Just Work. Then a New Edge Node Couldn't Pull Images","Kubo Team",{"type":8,"value":9,"toc":333},"minimark",[10,15,33,40,46,57,61,72,78,95,108,117,121,137,143,149,156,160,180,186,206,210,238,244,252,256,265,271,274,302,311,315,321],[11,12,14],"h2",{"id":13},"_1-the-day-only-that-one-node-failed-to-pull-images","1. The Day Only \"That One Node\" Failed to Pull Images",[16,17,18,19,23,24,28,29,32],"p",{},"K3s ships with a feature called the ",[20,21,22],"strong",{},"embedded registry mirror",", which is supposed to automatically share images across the entire cluster. When our edge footprint grew and we needed to add one more K3s agent node at a new factory site, we assumed this feature meant no extra configuration was needed. The procedure was the same as for any other node: just join the cluster with ",[25,26,27],"code",{},"k3s agent",". A few minutes later, only the Pods scheduled on that new node got stuck in ",[25,30,31],{},"ImagePullBackOff",".",[16,34,35],{},[36,37],"img",{"alt":38,"src":39},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Fsection01.webp",[16,41,42,43,45],{},"The existing nodes were pulling the exact same image without any problem. Only the new node was affected. We first suspected network connectivity, but both ping and DNS resolution were fine. The real cause was something far more mundane: the specification of K3s's ",[20,44,22],{}," itself.",[16,47,48,49,56],{},"Since v1.27, K3s has bundled ",[50,51,55],"a",{"href":52,"rel":53},"https:\u002F\u002Fgithub.com\u002Fspegel-org\u002Fspegel",[54],"nofollow","Spegel",", a stateless peer-to-peer OCI registry mirror. Assuming that \"embedded\" meant any new node would automatically benefit from it the moment it joined turned out to be our first mistake.",[11,58,60],{"id":59},"_2-what-k3ss-embedded-registry-mirror-spegel-actually-does","2. What K3s's Embedded Registry Mirror (Spegel) Actually Does",[16,62,63,64,67,68,71],{},"Spegel lets any node in the cluster pull images directly from another node over P2P, as long as that image already exists somewhere in the cluster. Enabling it is as simple as starting server nodes with the ",[25,65,66],{},"--embedded-registry"," flag, or setting ",[25,69,70],{},"embedded-registry: true"," in the configuration file.",[16,73,74],{},[36,75],{"alt":76,"src":77},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Fsection02.webp",[16,79,80,81,86,87,90,91,94],{},"That simplicity comes with network requirements, though. According to ",[50,82,85],{"href":83,"rel":84},"https:\u002F\u002Fdocs.k3s.io\u002Finstallation\u002Fregistry-mirror",[54],"K3s's official documentation",", every node must be able to reach every other node over their internal IP addresses on TCP ports ",[20,88,89],{},"5001"," (P2P distribution) and ",[20,92,93],{},"6443"," (each node's local OCI registry). If connectivity fails, pulls fall back to the external registry, but with added latency.",[16,96,97,98,103,104,107],{},"Access to the registry API requires a client certificate signed by the cluster's CA, and the embedded registry itself is read-only. ",[50,99,102],{"href":100,"rel":101},"https:\u002F\u002Fdocumentation.suse.com\u002Fcloudnative\u002Fk3s\u002Flatest\u002Fen\u002Finstallation\u002Fregistry-mirror.html",[54],"SUSE's official documentation"," also makes clear that this distributed registry \"assumes an equal level of privilege and trust between all cluster members.\" In other words, once a registry is listed as a mirror target in ",[25,105,106],{},"registries.yaml",", its authenticated images become accessible from every node without further credential verification.",[16,109,110,111,116],{},"With a fully managed service like ",[50,112,115],{"href":113,"rel":114},"https:\u002F\u002Fkubo.hexabase.io\u002F",[54],"Kubo",", this kind of network requirement and TLS configuration is validated by the platform itself, but in a self-managed setup you need to verify it manually.",[11,118,120],{"id":119},"_3-root-cause-1-every-node-required-is-an-easy-spec-to-miss","3. Root Cause #1: \"Every Node Required\" Is an Easy Spec to Miss",[16,122,123,124,127,128,133,134,32],{},"As we dug further, the first root cause emerged. K3s checks for the existence of ",[25,125,126],{},"\u002Fetc\u002Francher\u002Fk3s\u002Fregistries.yaml"," at node startup, and if it exists, applies it to the containerd configuration. ",[50,129,132],{"href":130,"rel":131},"https:\u002F\u002Fdocs.k3s.io\u002Finstallation\u002Fprivate-registry",[54],"K3s's official private registry documentation"," explicitly states that this configuration file ",[20,135,136],{},"must be placed on every node, both server and agent, that needs to pull images from a private registry",[16,138,139],{},[36,140],{"alt":141,"src":142},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Fsection03.webp",[16,144,145,146,148],{},"Our provisioning runbook for new edge nodes covered OS setup and installing the K3s agent, but the step to copy ",[25,147,106],{}," was missing. Because existing nodes had the file placed manually back when the cluster was first built, the gap never surfaced — it had quietly become an \"implicit assumption\" that got forgotten every time a new node was added.",[16,150,151,152,155],{},"What makes this even trickier is that even after placing the file, ",[20,153,154],{},"the change only takes effect after restarting K3s on that node",". Leave the drift unnoticed without a restart, and it never resolves itself.",[11,157,159],{"id":158},"_4-root-cause-2-wildcard-mirrors-and-a-containerd-20-regression","4. Root Cause #2: Wildcard Mirrors and a containerd 2.0 Regression",[16,161,162,163,165,166,169,170,175,176,179],{},"Even after fixing root cause #1, things were still unstable — because our ",[25,164,106],{}," mirror configuration used a wildcard, ",[25,167,168],{},"\"*\":",". It's a convenient way to enable distributed mirroring for every registry at once, but as reported in ",[50,171,174],{"href":172,"rel":173},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F11857",[54],"k3s-io\u002Fk3s Issue #11857",", K3s versions from v1.32.2+k3s1 onward (which moved to containerd 2.0) had a regression where fallback failed to work correctly when an image wasn't available in the mirror, resulting in a ",[25,177,178],{},"not found"," pull failure.",[16,181,182],{},[36,183],{"alt":184,"src":185},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Fsection04.webp",[16,187,188,189,191,192,197,198,201,202,205],{},"The bug was fixed in v1.31.7-rc1 and v1.32.3-rc2, but the combination of the wildcard setting and the missing ",[25,190,106],{}," made isolating the actual cause far harder than it should have been. The workaround is to avoid the wildcard and explicitly list the target registries instead. Separately, ",[50,193,196],{"href":194,"rel":195},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F10246",[54],"Issue #10246"," documents another known bug where embedded registry pulls fail when ",[25,199,200],{},"--bind-address"," is set to anything other than the default ",[25,203,204],{},"127.0.0.1"," — worth checking if you're running a custom network configuration.",[11,207,209],{"id":208},"_5-root-cause-3-credential-drift-between-nodes-fails-silently","5. Root Cause #3: Credential Drift Between Nodes Fails Silently",[16,211,212,213,215,216,219,220,222,223,226,227,226,230,233,234,237],{},"After distributing ",[25,214,106],{}," and cleaning up the wildcard, one problem remained: a mismatch in the private registry credentials themselves. The ",[25,217,218],{},"configs"," section of ",[25,221,106],{}," supports basic auth or TLS certificates (",[25,224,225],{},"cert_file",", ",[25,228,229],{},"key_file",[25,231,232],{},"ca_file","), but if these values differ slightly between nodes, the result is a ",[20,235,236],{},"silent failure"," with an ambiguous error message.",[16,239,240],{},[36,241],{"alt":242,"src":243},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Fsection05.webp",[16,245,246,251],{},[50,247,250],{"href":248,"rel":249},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-02-09-k3s-registry-mirror-air-gapped\u002Fview",[54],"An operational report from OneUptime"," points out exactly this pattern: when nodes are provisioned at different times, image bundles and credentials can drift out of sync. Because authentication failures don't leave logs as clear as a network connectivity failure, they tend to get deprioritized during troubleshooting — but in air-gapped operations, they're one of the first things worth suspecting.",[11,253,255],{"id":254},"_6-how-this-differs-from-vanilla-kubernetes-and-the-right-way-to-distribute-configuration","6. How This Differs From Vanilla Kubernetes, and the Right Way to Distribute Configuration",[16,257,258,259,264],{},"It's worth restating the bigger picture: a built-in P2P registry mirror like Spegel simply doesn't exist in vanilla, standard Kubernetes. Kubernetes itself has no registry functionality — the kubelet just pulls directly from whatever registry is configured — so building an equivalent capability means standing up and operating your own pull-through cache, such as Harbor. ",[50,260,263],{"href":261,"rel":262},"https:\u002F\u002Fdocs.rke2.io\u002Finstall\u002Fregistry_mirror",[54],"RKE2's official documentation"," confirms the same Spegel-based mechanism is used there too, showing this is a shared design philosophy across the K3s family of distributions.",[16,266,267],{},[36,268],{"alt":269,"src":270},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Fsection06.webp",[16,272,273],{},"Based on what we learned, the correct distribution workflow when adding nodes in an air-gapped or edge environment looks like this:",[275,276,277,287,296],"ul",{},[278,279,280,283,284,286],"li",{},[20,281,282],{},"Automate distribution",": bake ",[25,285,106],{}," into your Ansible or cloud-init templates so there's no manual copy step to forget",[278,288,289,292,293,295],{},[20,290,291],{},"Avoid wildcards",": list target registries explicitly instead of using ",[25,294,168],{},", to avoid containerd version-dependent fallback behavior",[278,297,298,301],{},[20,299,300],{},"Add a pre-flight health check",": include a step in your node-join playbook that verifies reachability on ports 5001 and 6443, and confirms credentials match across nodes",[16,303,304,305,310],{},"This kind of distribution and verification automation is exactly the operational burden that grows heavier as your edge footprint expands. ",[50,306,309],{"href":307,"rel":308},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[54],"Kubo On-Premise"," is built around full self-management in air-gapped environments, and can take on this registry distribution and certificate management work as a fully managed service.",[11,312,314],{"id":313},"summary","Summary",[16,316,317,318,320],{},"The root of this incident was the assumption that \"it's a built-in feature, so it must automatically work on every node.\" In reality, K3s's embedded registry mirror (Spegel) is powerful, but it comes bundled with three distinct pitfalls: the all-nodes requirement for ",[25,319,106],{},", a known wildcard-mirror regression bug, and silent credential drift between nodes.",[16,322,323,324,327,328,32],{},"Re-running this same investigation every time you add an edge site isn't realistic. If you're scaling edge-plus-Kubernetes operations while carrying data-sovereignty or air-gapped requirements, it's worth considering a ",[50,325,309],{"href":307,"rel":326},[54]," setup that hands registry distribution and certificate management off to the platform. You can start the conversation via ",[50,329,332],{"href":330,"rel":331},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[54],"Contact Us",{"title":334,"searchDepth":335,"depth":335,"links":336},"",2,[337,338,339,340,341,342,343],{"id":13,"depth":335,"text":14},{"id":59,"depth":335,"text":60},{"id":119,"depth":335,"text":120},{"id":158,"depth":335,"text":159},{"id":208,"depth":335,"text":209},{"id":254,"depth":335,"text":255},{"id":313,"depth":335,"text":314},"2026-10-30","K3s's embedded registry mirror (Spegel) is supposed to work cluster-wide, but a newly added edge node alone failed to pull images. The cause: an undistributed registries.yaml, credential drift, and a known containerd 2.0 bug. Here's the correct distribution workflow for air-gapped operations.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join",{"title":5,"description":345},"blog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join",[355,356,357,358,359,360],"k3s","kubernetes","spegel","registry-mirror","airgap","edge-computing","nvOmWRIAzIq0L243hgUmSyKY1-CnPxDMliicdc_zXKE",[363,371,379,387,396,404],{"path":364,"title":365,"description":366,"date":367,"tags":368},"\u002Fblog\u002Fen\u002Fk3s-prometheus-agent-mode-stale-marker-edge-monitoring","We Switched K3s Edge Prometheus to Agent Mode. The Moment a Target Disappeared, So Did Its Metrics","Running Prometheus Agent mode on K3s edge clusters can leave stale values in your central dashboard when a target disappears, depending on how stale markers are handled. Here's how to design K3s edge monitoring correctly.","2026-11-03",[355,356,369,360,370],"prometheus","observability",{"path":372,"title":373,"description":374,"date":375,"tags":376},"\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain","The Plan Never Disappeared. Why K3s's system-upgrade-controller Got Stuck on One Node","We set up automated upgrades with K3s's system-upgrade-controller, and the Plan stopped progressing on a single node. The culprits were PodDisruptionBudgets and taints. Here's how the mechanism works, how to recover, and how Kubo helps.","2026-10-31",[355,356,377,378,360],"system-upgrade-controller","rolling-upgrade",{"path":380,"title":381,"description":382,"date":383,"tags":384},"\u002Fblog\u002Fen\u002Fk3s-microshift-etcd-ostree-edge-comparison","Is Pinning etcd at the Edge a Trap or a Trade-off? K3s vs MicroShift and the Real Cost of OSTree Auto-Rollback","K3s and MicroShift are both called \"lightweight Kubernetes,\" but their assumptions about datastores and OS update models are fundamentally different. We compare etcd lock-in and OSTree auto-rollback to help you choose for production edge deployments.","2026-10-29",[355,385,356,360,386],"microshift","redhat",{"path":388,"title":389,"description":390,"date":391,"tags":392},"\u002Fblog\u002Fen\u002Fk3s-vs-microk8s-ha-etcd-dqlite-edge-production","K3s vs MicroK8s: Which One Actually Wins in Production? The HA Fork Between dqlite and etcd","K3s vs MicroK8s: both are called 'lightweight Kubernetes,' but the real production divide is HA architecture. We compare K3s's embedded etcd and MicroK8s's dqlite using official documentation data.","2026-10-25",[355,393,356,394,360,395],"microk8s","high-availability","comparison",{"path":397,"title":398,"description":399,"date":400,"tags":401},"\u002Fblog\u002Fen\u002Fedge-k3s-hardware-selection-beyond-appliance","Homelabbers Are Ditching ISP Routers for Self-Built PCs. Edge Kubernetes Should Rethink Its Appliance Habit Too","A growing number of individuals are replacing ISP-supplied routers with self-built mini PCs running open-source firmware. The same logic applies to edge Kubernetes: a hardware selection checklist and a GitOps operating model for standardizing dedicated appliances out of the fleet.","2026-08-29",[355,356,360,402,403],"hardware-selection","network-appliance",{"path":405,"title":406,"description":407,"date":408,"tags":409},"\u002Fblog\u002Fen\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium","Bare-Metal K3s LoadBalancer: When ARP Hits Its Limit, Design for BGP","When you can't rely on a cloud load balancer for on-prem or edge K3s clusters, why does Layer2 (ARP) load balancing eventually hit a wall? A practical look at MetalLB's BGP mode vs. Cilium's BGP Control Plane, and how to decide when to migrate.","2026-08-26",[355,356,410,411,412,360],"metallb","bgp","load-balancer",1793798727213]