Your K3s DNS Fails a Few Times a Day. The Culprit Isn't ndots:5 or conntrack — It's Both, Multiplied

"The API call that failed a minute ago just worked fine on retry."

A request to an external API times out a few times a day. The logs show nothing unusual. Retry it, and it succeeds as if nothing happened. If you run K3s in production, you've probably run into this intermittent DNS failure at some point.

The tricky part is that the symptom can't be explained by a single root cause. Many teams burn hours guessing "maybe CoreDNS is overloaded," when what's actually at play is a "multiplication" of two unrelated problems: DNS query amplification caused by ndots:5 and a kernel-level UDP conntrack race condition. On top of that, NodeLocal DNSCache — the very fix teams deploy to solve this — can itself become a new source of intermittent failures under certain conditions.

This article breaks down the real causes of DNS intermittency on K3s clusters, layer by layer. These traps are the kind of thing that should be designed out at the cluster level, not learned the hard way by an ops team one incident at a time (we'll touch on how the managed Kubernetes service Kubo addresses this at the end).

One Lookup, Four Queries: How ndots:5 Quietly Multiplies Your DNS Traffic

The first thing worth suspecting is ndots:5, an option automatically set in every pod's /etc/resolv.conf. As explained in Kubenatives' write-up, this setting tells the resolver: if a name has fewer than 5 dots, append the search domains first and try those before trying the name as-is.

Here's what that means in practice. When a pod tries to resolve an external domain like api.stripe.com (only 2 dots), the queries actually go out in this order:

1. api.stripe.com.default.svc.cluster.local → NXDOMAIN
2. api.stripe.com.svc.cluster.local → NXDOMAIN
3. api.stripe.com.cluster.local → NXDOMAIN
4. api.stripe.com → success

In other words, what looks like a single lookup is actually four queries, and three of them are guaranteed to fail by design. It's easy to miss on a low-traffic cluster, but as the number of pods grows, these wasted queries pile up and push both CoreDNS load and latency higher.

Fix: use a trailing-dot FQDN, or lower ndots

For applications that talk to a lot of external domains, either of these reduces the wasted queries:

• Specify the destination as an explicit FQDN with a trailing dot (api.stripe.com.), which tells the resolver "this is already a complete name, don't append search domains"
• Lower ndots to something like "2" via the pod's dnsConfig (verify beforehand that this doesn't break resolution of internal service names)

The Second Trap: A Kernel-Level conntrack Race That Produces a "5-Second Wait"

Query amplification from ndots:5 is already annoying enough on its own, but K3s/Kubernetes clusters have a second, kernel-level problem layered on top. As reported in weaveworks/weave Issue #3287, this is a kernel-level SNAT race condition.

Most resolvers query A (IPv4) and AAAA (IPv6) records simultaneously. When these two UDP packets get NAT-translated using the same source port, they can collide in the Linux kernel's conntrack table. If one packet gets dropped, the application waits out a timeout (often 5 seconds) before retrying — and only then does it succeed. This is a known issue that isn't specific to any one CNI (Weave, Flannel, and others all hit it), and it's referenced both in Kubernetes' own NodeLocalDNS documentation and in kubernetes/kubernetes#56903.

The official documentation also points out that UDP DNS queries continuously creating conntrack entries is itself a risk, noting that moving DNS queries from UDP to TCP "would reduce tail latency attributed to dropped UDP packets and DNS timeouts usually up to 30s." UDP conntrack entries stick around in the table until they time out, so the busier a cluster's DNS traffic, the more pressure builds up on that table.

Fix: serialize A/AAAA lookups with single-request-reopen

OneUptime's debugging guide describes a workaround for kernels older than 5.9: the following pod setting sends A and AAAA queries sequentially instead of simultaneously, which reduces contention on the same socket and lowers the odds of hitting the 5-second wait.

dnsConfig:
  options:
    - name: single-request-reopen

The Fix Becomes a Trap: How NodeLocal DNSCache Creates a New Kind of Intermittent Failure

The standard prescription for both problems above is NodeLocal DNSCache. It runs a caching agent as a DaemonSet on every node, listening on a link-local address (in the 169.254.0.0/16 range, per the official Kubernetes documentation), so that cache hits can skip the network hop to CoreDNS entirely. One operations blog reported an incident where a mass pod startup caused a surge of DNS lookups, and measured a latency improvement after adopting NodeLocal DNSCache — from hundreds of milliseconds down to single-digit milliseconds.

But that's not the end of the story. Because NodeLocal DNSCache is itself a DaemonSet, it can become a new source of intermittent failures, something that's easy to overlook. ADHDecode's write-up points to two specific scenarios:

• During a rolling update: updating the NodeLocal DNSCache DaemonSet temporarily leaves some nodes without a running cache instance
• Under node resource pressure: when CPU, memory, or disk pressure triggers pod eviction, NodeLocal DNSCache pods can be evicted too

What's particularly interesting is the shape of the symptom this produces: internal Kubernetes service names still resolve fine, but external domains occasionally time out — a seemingly contradictory pattern caused by internal and external names taking different resolution paths. If you've deployed NodeLocal DNSCache and are still seeing occasional failures, it's worth checking whether the cache pod itself is actually running on the node in question.

Keeping constant watch over this DaemonSet's rollout timing and node resource pressure is the kind of unglamorous operational work that quietly eats up engineering time. Kubo, a managed Kubernetes service built on K3s, surfaces the health of standard components like NodeLocal DNSCache in Captain UI and manages the rollover during updates on the operations side.

K3s's Own Trap: Why kubectl edit configmap coredns Doesn't Stick

Even understanding all three problems above isn't enough — there's one more trap specific to K3s. K3s manages CoreDNS as a bundled Addon, and the official K3s documentation states that to customize the CoreDNS configuration, you create a ConfigMap named coredns-custom in the kube-system namespace.

In other words, if you directly edit the config with kubectl edit configmap coredns, K3s redeploys CoreDNS as an Addon on startup and your change disappears. OneUptime's K3s-specific guide lays out the correct persistence procedure:

• Create a ConfigMap named coredns-custom
• Keys matching *.override get imported into the default :.53 server block
• Keys matching *.server are treated as additional server blocks
• Placing this ConfigMap's manifest at /var/lib/rancher/k3s/server/manifests/coredns-custom.yaml means the configuration survives K3s restarts and upgrades

The official K3s documentation also notes that the --resolv-conf option lets you pass an alternative resolver configuration without modifying the system's own config files. Anyone customizing DNS on a K3s cluster needs to know this persistence pattern up front.

Intermittent DNS Failures Don't Have "One Root Cause" — That's the Point

DNS intermittency on a K3s cluster is never the work of a single culprit. ndots:5 amplifies name resolution into four times the queries; a UDP conntrack kernel race turns packet loss into a 5-second wait; the fix for both, NodeLocal DNSCache, can itself become a new failure source during rolling updates or node pressure; and K3s's own Addon management wipes out naive config changes on every restart. These four layers are independent of each other, yet they compound — which is exactly why treating any single one as "the fix" tends to leave the problem to resurface later.

Learning these traps one incident at a time isn't where infrastructure expertise should be spent. Kubo, a managed Kubernetes service built on K3s, ships clusters with sensible NodeLocal DNSCache configuration and CoreDNS customization persistence already built in. Before building and validating a DNS stack like this from scratch, it's worth comparing against managed options — including the pricing. Kubo is designed to combine the convenience of a fully managed service like EKS/AKS with the cost efficiency that comes from being built on K3s. If data sovereignty on-premises is a requirement, Kubo On-Premise is also an option. For questions about cluster design, including the DNS stack, feel free to reach out via contact us.