1. The Day Only "That One Node" Failed to Pull Images
K3s ships with a feature called the embedded registry mirror, which is supposed to automatically share images across the entire cluster. When our edge footprint grew and we needed to add one more K3s agent node at a new factory site, we assumed this feature meant no extra configuration was needed. The procedure was the same as for any other node: just join the cluster with k3s agent. A few minutes later, only the Pods scheduled on that new node got stuck in ImagePullBackOff.
The existing nodes were pulling the exact same image without any problem. Only the new node was affected. We first suspected network connectivity, but both ping and DNS resolution were fine. The real cause was something far more mundane: the specification of K3s's embedded registry mirror itself.
Since v1.27, K3s has bundled Spegel, a stateless peer-to-peer OCI registry mirror. Assuming that "embedded" meant any new node would automatically benefit from it the moment it joined turned out to be our first mistake.
2. What K3s's Embedded Registry Mirror (Spegel) Actually Does
Spegel lets any node in the cluster pull images directly from another node over P2P, as long as that image already exists somewhere in the cluster. Enabling it is as simple as starting server nodes with the --embedded-registry flag, or setting embedded-registry: true in the configuration file.
That simplicity comes with network requirements, though. According to K3s's official documentation, every node must be able to reach every other node over their internal IP addresses on TCP ports 5001 (P2P distribution) and 6443 (each node's local OCI registry). If connectivity fails, pulls fall back to the external registry, but with added latency.
Access to the registry API requires a client certificate signed by the cluster's CA, and the embedded registry itself is read-only. SUSE's official documentation also makes clear that this distributed registry "assumes an equal level of privilege and trust between all cluster members." In other words, once a registry is listed as a mirror target in registries.yaml, its authenticated images become accessible from every node without further credential verification.
With a fully managed service like Kubo, this kind of network requirement and TLS configuration is validated by the platform itself, but in a self-managed setup you need to verify it manually.
3. Root Cause #1: "Every Node Required" Is an Easy Spec to Miss
As we dug further, the first root cause emerged. K3s checks for the existence of /etc/rancher/k3s/registries.yaml at node startup, and if it exists, applies it to the containerd configuration. K3s's official private registry documentation explicitly states that this configuration file must be placed on every node, both server and agent, that needs to pull images from a private registry.
Our provisioning runbook for new edge nodes covered OS setup and installing the K3s agent, but the step to copy registries.yaml was missing. Because existing nodes had the file placed manually back when the cluster was first built, the gap never surfaced — it had quietly become an "implicit assumption" that got forgotten every time a new node was added.
What makes this even trickier is that even after placing the file, the change only takes effect after restarting K3s on that node. Leave the drift unnoticed without a restart, and it never resolves itself.
4. Root Cause #2: Wildcard Mirrors and a containerd 2.0 Regression
Even after fixing root cause #1, things were still unstable — because our registries.yaml mirror configuration used a wildcard, "*":. It's a convenient way to enable distributed mirroring for every registry at once, but as reported in k3s-io/k3s Issue #11857, K3s versions from v1.32.2+k3s1 onward (which moved to containerd 2.0) had a regression where fallback failed to work correctly when an image wasn't available in the mirror, resulting in a not found pull failure.
The bug was fixed in v1.31.7-rc1 and v1.32.3-rc2, but the combination of the wildcard setting and the missing registries.yaml made isolating the actual cause far harder than it should have been. The workaround is to avoid the wildcard and explicitly list the target registries instead. Separately, Issue #10246 documents another known bug where embedded registry pulls fail when --bind-address is set to anything other than the default 127.0.0.1 — worth checking if you're running a custom network configuration.
5. Root Cause #3: Credential Drift Between Nodes Fails Silently
After distributing registries.yaml and cleaning up the wildcard, one problem remained: a mismatch in the private registry credentials themselves. The configs section of registries.yaml supports basic auth or TLS certificates (cert_file, key_file, ca_file), but if these values differ slightly between nodes, the result is a silent failure with an ambiguous error message.
An operational report from OneUptime points out exactly this pattern: when nodes are provisioned at different times, image bundles and credentials can drift out of sync. Because authentication failures don't leave logs as clear as a network connectivity failure, they tend to get deprioritized during troubleshooting — but in air-gapped operations, they're one of the first things worth suspecting.
6. How This Differs From Vanilla Kubernetes, and the Right Way to Distribute Configuration
It's worth restating the bigger picture: a built-in P2P registry mirror like Spegel simply doesn't exist in vanilla, standard Kubernetes. Kubernetes itself has no registry functionality — the kubelet just pulls directly from whatever registry is configured — so building an equivalent capability means standing up and operating your own pull-through cache, such as Harbor. RKE2's official documentation confirms the same Spegel-based mechanism is used there too, showing this is a shared design philosophy across the K3s family of distributions.
Based on what we learned, the correct distribution workflow when adding nodes in an air-gapped or edge environment looks like this:
- Automate distribution: bake
registries.yamlinto your Ansible or cloud-init templates so there's no manual copy step to forget - Avoid wildcards: list target registries explicitly instead of using
"*":, to avoid containerd version-dependent fallback behavior - Add a pre-flight health check: include a step in your node-join playbook that verifies reachability on ports 5001 and 6443, and confirms credentials match across nodes
This kind of distribution and verification automation is exactly the operational burden that grows heavier as your edge footprint expands. Kubo On-Premise is built around full self-management in air-gapped environments, and can take on this registry distribution and certificate management work as a fully managed service.
Summary
The root of this incident was the assumption that "it's a built-in feature, so it must automatically work on every node." In reality, K3s's embedded registry mirror (Spegel) is powerful, but it comes bundled with three distinct pitfalls: the all-nodes requirement for registries.yaml, a known wildcard-mirror regression bug, and silent credential drift between nodes.
Re-running this same investigation every time you add an edge site isn't realistic. If you're scaling edge-plus-Kubernetes operations while carrying data-sovereignty or air-gapped requirements, it's worth considering a Kubo On-Premise setup that hands registry distribution and certificate management off to the platform. You can start the conversation via Contact Us.