[{"data":1,"prerenderedAt":546},["ShallowReactive",2],{"blog-en-k3s-dns-intermittent-ndots-conntrack-race":3,"blog-related-en-k3s-dns-intermittent-ndots-conntrack-race":485,"blog-en-k3s-dns-intermittent-ndots-conntrack-race-alt":473},{"id":4,"title":5,"author":6,"body":7,"date":467,"description":468,"extension":469,"image":470,"locale":471,"meta":472,"navigation":473,"path":474,"seo":475,"stem":476,"tags":477,"__hash__":484},"blog\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race.md","Your K3s DNS Fails a Few Times a Day. The Culprit Isn't ndots:5 or conntrack — It's Both, Multiplied","Kubo Team",{"type":8,"value":9,"toc":454},"minimark",[10,15,23,31,46,57,61,67,85,92,117,124,129,132,157,161,167,180,194,197,201,209,255,259,265,286,299,313,316,323,331,352,364,398,405,409,426,450],[11,12,14],"h2",{"id":13},"the-api-call-that-failed-a-minute-ago-just-worked-fine-on-retry","\"The API call that failed a minute ago just worked fine on retry.\"",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race\u002Fsection01.webp",[16,24,25,26,30],{},"A request to an external API times out a few times a day. The logs show nothing unusual. Retry it, and it succeeds as if nothing happened. If you run K3s in production, you've probably run into this ",[27,28,29],"strong",{},"intermittent DNS failure"," at some point.",[16,32,33,34,37,38,41,42,45],{},"The tricky part is that the symptom can't be explained by a single root cause. Many teams burn hours guessing \"maybe CoreDNS is overloaded,\" when what's actually at play is a \"multiplication\" of two unrelated problems: ",[27,35,36],{},"DNS query amplification caused by ndots:5"," and a ",[27,39,40],{},"kernel-level UDP conntrack race condition",". On top of that, ",[27,43,44],{},"NodeLocal DNSCache"," — the very fix teams deploy to solve this — can itself become a new source of intermittent failures under certain conditions.",[16,47,48,49,56],{},"This article breaks down the real causes of DNS intermittency on K3s clusters, layer by layer. These traps are the kind of thing that should be designed out at the cluster level, not learned the hard way by an ops team one incident at a time (we'll touch on how the managed Kubernetes service ",[50,51,55],"a",{"href":52,"rel":53},"https:\u002F\u002Fkubo.hexabase.io\u002F",[54],"nofollow","Kubo"," addresses this at the end).",[11,58,60],{"id":59},"one-lookup-four-queries-how-ndots5-quietly-multiplies-your-dns-traffic","One Lookup, Four Queries: How ndots:5 Quietly Multiplies Your DNS Traffic",[16,62,63],{},[19,64],{"alt":65,"src":66},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race\u002Fsection02.webp",[16,68,69,70,74,75,78,79,84],{},"The first thing worth suspecting is ",[71,72,73],"code",{},"ndots:5",", an option automatically set in every pod's ",[71,76,77],{},"\u002Fetc\u002Fresolv.conf",". As explained in ",[50,80,83],{"href":81,"rel":82},"https:\u002F\u002Fwww.kubenatives.com\u002Fp\u002Fkubernetes-dns-troubleshooting-coredns",[54],"Kubenatives' write-up",", this setting tells the resolver: if a name has fewer than 5 dots, append the search domains first and try those before trying the name as-is.",[16,86,87,88,91],{},"Here's what that means in practice. When a pod tries to resolve an external domain like ",[71,89,90],{},"api.stripe.com"," (only 2 dots), the queries actually go out in this order:",[93,94,95,102,107,112],"ol",{},[96,97,98,101],"li",{},[71,99,100],{},"api.stripe.com.default.svc.cluster.local"," → NXDOMAIN",[96,103,104,101],{},[71,105,106],{},"api.stripe.com.svc.cluster.local",[96,108,109,101],{},[71,110,111],{},"api.stripe.com.cluster.local",[96,113,114,116],{},[71,115,90],{}," → success",[16,118,119,120,123],{},"In other words, ",[27,121,122],{},"what looks like a single lookup is actually four queries, and three of them are guaranteed to fail"," by design. It's easy to miss on a low-traffic cluster, but as the number of pods grows, these wasted queries pile up and push both CoreDNS load and latency higher.",[125,126,128],"h3",{"id":127},"fix-use-a-trailing-dot-fqdn-or-lower-ndots","Fix: use a trailing-dot FQDN, or lower ndots",[16,130,131],{},"For applications that talk to a lot of external domains, either of these reduces the wasted queries:",[133,134,135,142],"ul",{},[96,136,137,138,141],{},"Specify the destination as an explicit FQDN with a trailing dot (",[71,139,140],{},"api.stripe.com.","), which tells the resolver \"this is already a complete name, don't append search domains\"",[96,143,144,145,148,149,152,153,156],{},"Lower ",[71,146,147],{},"ndots"," to something like ",[71,150,151],{},"\"2\""," via the pod's ",[71,154,155],{},"dnsConfig"," (verify beforehand that this doesn't break resolution of internal service names)",[11,158,160],{"id":159},"the-second-trap-a-kernel-level-conntrack-race-that-produces-a-5-second-wait","The Second Trap: A Kernel-Level conntrack Race That Produces a \"5-Second Wait\"",[16,162,163],{},[19,164],{"alt":165,"src":166},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race\u002Fsection03.webp",[16,168,169,170,175,176,179],{},"Query amplification from ndots:5 is already annoying enough on its own, but K3s\u002FKubernetes clusters have a second, kernel-level problem layered on top. As reported in ",[50,171,174],{"href":172,"rel":173},"https:\u002F\u002Fgithub.com\u002Fweaveworks\u002Fweave\u002Fissues\u002F3287",[54],"weaveworks\u002Fweave Issue #3287",", this is a ",[27,177,178],{},"kernel-level SNAT race condition",".",[16,181,182,183,188,189,179],{},"Most resolvers query A (IPv4) and AAAA (IPv6) records simultaneously. When these two UDP packets get NAT-translated using the same source port, they can collide in the Linux kernel's conntrack table. If one packet gets dropped, the application waits out a timeout (often 5 seconds) before retrying — and only then does it succeed. This is a known issue that isn't specific to any one CNI (Weave, Flannel, and others all hit it), and it's referenced both in ",[50,184,187],{"href":185,"rel":186},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fadminister-cluster\u002Fnodelocaldns\u002F",[54],"Kubernetes' own NodeLocalDNS documentation"," and in ",[50,190,193],{"href":191,"rel":192},"https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fkubernetes\u002Fissues\u002F56903",[54],"kubernetes\u002Fkubernetes#56903",[16,195,196],{},"The official documentation also points out that UDP DNS queries continuously creating conntrack entries is itself a risk, noting that moving DNS queries from UDP to TCP \"would reduce tail latency attributed to dropped UDP packets and DNS timeouts usually up to 30s.\" UDP conntrack entries stick around in the table until they time out, so the busier a cluster's DNS traffic, the more pressure builds up on that table.",[125,198,200],{"id":199},"fix-serialize-aaaaa-lookups-with-single-request-reopen","Fix: serialize A\u002FAAAA lookups with single-request-reopen",[16,202,203,208],{},[50,204,207],{"href":205,"rel":206},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-01-06-debug-kubernetes-dns-issues\u002Fview",[54],"OneUptime's debugging guide"," describes a workaround for kernels older than 5.9: the following pod setting sends A and AAAA queries sequentially instead of simultaneously, which reduces contention on the same socket and lowers the odds of hitting the 5-second wait.",[210,211,216],"pre",{"className":212,"code":213,"language":214,"meta":215,"style":215},"language-yaml shiki shiki-themes tokyo-night","dnsConfig:\n  options:\n    - name: single-request-reopen\n","yaml","",[71,217,218,230,238],{"__ignoreMap":215},[219,220,223,226],"span",{"class":221,"line":222},"line",1,[219,224,155],{"class":225},"s0U2E",[219,227,229],{"class":228},"sAklC",":\n",[219,231,233,236],{"class":221,"line":232},2,[219,234,235],{"class":225},"  options",[219,237,229],{"class":228},[219,239,241,245,248,251],{"class":221,"line":240},3,[219,242,244],{"class":243},"sgJMe","    -",[219,246,247],{"class":225}," name",[219,249,250],{"class":228},":",[219,252,254],{"class":253},"sPY7s"," single-request-reopen\n",[11,256,258],{"id":257},"the-fix-becomes-a-trap-how-nodelocal-dnscache-creates-a-new-kind-of-intermittent-failure","The Fix Becomes a Trap: How NodeLocal DNSCache Creates a New Kind of Intermittent Failure",[16,260,261],{},[19,262],{"alt":263,"src":264},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race\u002Fsection04.webp",[16,266,267,268,270,271,274,275,279,280,285],{},"The standard prescription for both problems above is ",[27,269,44],{},". It runs a caching agent as a DaemonSet on every node, listening on a link-local address (in the ",[71,272,273],{},"169.254.0.0\u002F16"," range, per the ",[50,276,278],{"href":185,"rel":277},[54],"official Kubernetes documentation","), so that cache hits can skip the network hop to CoreDNS entirely. One operations blog ",[50,281,284],{"href":282,"rel":283},"https:\u002F\u002Fsvalle.ru\u002Fposts\u002Fkubernetes\u002Fcoredns-nodelocal-cache\u002F",[54],"reported"," an incident where a mass pod startup caused a surge of DNS lookups, and measured a latency improvement after adopting NodeLocal DNSCache — from hundreds of milliseconds down to single-digit milliseconds.",[16,287,288,289,292,293,298],{},"But that's not the end of the story. Because NodeLocal DNSCache is itself a DaemonSet, ",[27,290,291],{},"it can become a new source of intermittent failures",", something that's easy to overlook. ",[50,294,297],{"href":295,"rel":296},"https:\u002F\u002Fadhdecode.com\u002Fdebugging\u002Fcoredns\u002Fnode-local-dns-upstream-timeout\u002F",[54],"ADHDecode's write-up"," points to two specific scenarios:",[133,300,301,307],{},[96,302,303,306],{},[27,304,305],{},"During a rolling update",": updating the NodeLocal DNSCache DaemonSet temporarily leaves some nodes without a running cache instance",[96,308,309,312],{},[27,310,311],{},"Under node resource pressure",": when CPU, memory, or disk pressure triggers pod eviction, NodeLocal DNSCache pods can be evicted too",[16,314,315],{},"What's particularly interesting is the shape of the symptom this produces: internal Kubernetes service names still resolve fine, but external domains occasionally time out — a seemingly contradictory pattern caused by internal and external names taking different resolution paths. If you've deployed NodeLocal DNSCache and are still seeing occasional failures, it's worth checking whether the cache pod itself is actually running on the node in question.",[16,317,318,319,322],{},"Keeping constant watch over this DaemonSet's rollout timing and node resource pressure is the kind of unglamorous operational work that quietly eats up engineering time. ",[50,320,55],{"href":52,"rel":321},[54],", a managed Kubernetes service built on K3s, surfaces the health of standard components like NodeLocal DNSCache in Captain UI and manages the rollover during updates on the operations side.",[11,324,326,327,330],{"id":325},"k3ss-own-trap-why-kubectl-edit-configmap-coredns-doesnt-stick","K3s's Own Trap: Why ",[71,328,329],{},"kubectl edit configmap coredns"," Doesn't Stick",[16,332,333,334,337,338,343,344,347,348,351],{},"Even understanding all three problems above isn't enough — there's one more trap specific to K3s. K3s manages CoreDNS as a ",[27,335,336],{},"bundled Addon",", and the ",[50,339,342],{"href":340,"rel":341},"https:\u002F\u002Fdocs.k3s.io\u002Fadvanced",[54],"official K3s documentation"," states that to customize the CoreDNS configuration, you create a ConfigMap named ",[71,345,346],{},"coredns-custom"," in the ",[71,349,350],{},"kube-system"," namespace.",[16,353,354,355,357,358,363],{},"In other words, if you directly edit the config with ",[71,356,329],{},", K3s redeploys CoreDNS as an Addon on startup and your change disappears. ",[50,359,362],{"href":360,"rel":361},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-20-k3s-coredns-custom-config\u002Fview",[54],"OneUptime's K3s-specific guide"," lays out the correct persistence procedure:",[133,365,366,371,382,388],{},[96,367,368,369],{},"Create a ConfigMap named ",[71,370,346],{},[96,372,373,374,377,378,381],{},"Keys matching ",[71,375,376],{},"*.override"," get imported into the default ",[71,379,380],{},":.53"," server block",[96,383,373,384,387],{},[71,385,386],{},"*.server"," are treated as additional server blocks",[96,389,390,391,394,395],{},"Placing this ConfigMap's manifest at ",[71,392,393],{},"\u002Fvar\u002Flib\u002Francher\u002Fk3s\u002Fserver\u002Fmanifests\u002Fcoredns-custom.yaml"," means ",[27,396,397],{},"the configuration survives K3s restarts and upgrades",[16,399,400,401,404],{},"The official K3s documentation also notes that the ",[71,402,403],{},"--resolv-conf"," option lets you pass an alternative resolver configuration without modifying the system's own config files. Anyone customizing DNS on a K3s cluster needs to know this persistence pattern up front.",[11,406,408],{"id":407},"intermittent-dns-failures-dont-have-one-root-cause-thats-the-point","Intermittent DNS Failures Don't Have \"One Root Cause\" — That's the Point",[16,410,411,412,414,415,418,419,421,422,425],{},"DNS intermittency on a K3s cluster is never the work of a single culprit. ",[27,413,73],{}," amplifies name resolution into four times the queries; a ",[27,416,417],{},"UDP conntrack kernel race"," turns packet loss into a 5-second wait; the fix for both, ",[27,420,44],{},", can itself become a new failure source during rolling updates or node pressure; and ",[27,423,424],{},"K3s's own Addon management"," wipes out naive config changes on every restart. These four layers are independent of each other, yet they compound — which is exactly why treating any single one as \"the fix\" tends to leave the problem to resurface later.",[16,427,428,429,432,433,438,439,444,445,179],{},"Learning these traps one incident at a time isn't where infrastructure expertise should be spent. ",[50,430,55],{"href":52,"rel":431},[54],", a managed Kubernetes service built on K3s, ships clusters with sensible NodeLocal DNSCache configuration and CoreDNS customization persistence already built in. Before building and validating a DNS stack like this from scratch, it's worth comparing against managed options — including the ",[50,434,437],{"href":435,"rel":436},"https:\u002F\u002Fwww.hexabase.com\u002Fpricing\u002F",[54],"pricing",". Kubo is designed to combine the convenience of a fully managed service like EKS\u002FAKS with the cost efficiency that comes from being built on K3s. If data sovereignty on-premises is a requirement, ",[50,440,443],{"href":441,"rel":442},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[54],"Kubo On-Premise"," is also an option. For questions about cluster design, including the DNS stack, feel free to reach out via ",[50,446,449],{"href":447,"rel":448},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[54],"contact us",[451,452,453],"style",{},"html pre.shiki code .s0U2E, html code.shiki .s0U2E{--shiki-default:#F7768E}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sgJMe, html code.shiki .sgJMe{--shiki-default:#9ABDF5}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":215,"searchDepth":232,"depth":232,"links":455},[456,457,460,463,464,466],{"id":13,"depth":232,"text":14},{"id":59,"depth":232,"text":60,"children":458},[459],{"id":127,"depth":240,"text":128},{"id":159,"depth":232,"text":160,"children":461},[462],{"id":199,"depth":240,"text":200},{"id":257,"depth":232,"text":258},{"id":325,"depth":232,"text":465},"K3s's Own Trap: Why kubectl edit configmap coredns Doesn't Stick",{"id":407,"depth":232,"text":408},"2026-09-15","Intermittent DNS failures in K3s usually come from ndots:5 query amplification combined with a UDP conntrack race condition. We break down NodeLocal DNSCache's benefits and side effects, plus K3s's own coredns-custom persistence trap.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race",{"title":5,"description":468},"blog\u002Fen\u002Fk3s-dns-intermittent-ndots-conntrack-race",[478,479,480,481,482,483],"k3s","kubernetes","coredns","dns","conntrack","nodelocal-dnscache","_1DsmHRaZbQPW3zxLeLs9CKChObu--g10PXnrmvM-mM",[486,496,508,518,528,536],{"path":487,"title":488,"description":489,"date":490,"tags":491},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[478,479,492,493,494,495],"hpa","containerresource","autoscaling","service-mesh",{"path":497,"title":498,"description":499,"date":500,"tags":501},"\u002Fblog\u002Fen\u002Fk3s-vs-kubernetes-cni-flannel-default-comparison","K3s vs Standard Kubernetes: How Different Their CNI Starting Points Really Are","K3s ships with Flannel built in, while standard Kubernetes starts with no CNI selected at all. This comparison, grounded in the official K3s, Kubernetes, Calico, and Cilium docs, explains how that difference affects NetworkPolicy enforcement and network performance.","2026-10-20",[478,479,502,503,504,505,506,507],"cni","flannel","calico","cilium","networkpolicy","networking",{"path":509,"title":510,"description":511,"date":512,"tags":513},"\u002Fblog\u002Fen\u002Fk3s-topology-spread-constraints-poddisruptionbudget-node-drain-pending","We Drained a Node and a Pod Never Came Back. Why Topology Spread Constraints and PodDisruptionBudget Collide on K3s","Draining a node in production K3s left a Pod stuck Pending. The cause: Topology Spread Constraints' maxSkew colliding with PodDisruptionBudget. We verify how draining itself creates a node shortage, based on official documentation.","2026-10-19",[478,479,514,515,516,517],"topology-spread-constraints","poddisruptionbudget","scheduling","node-drain",{"path":519,"title":520,"description":521,"date":522,"tags":523},"\u002Fblog\u002Fen\u002Fkyverno-validatingadmissionpolicy-cel-migration-silent-bypass","Is the CEL Rewrite of Your Policy Really the Same? How Migrating from Kyverno to ValidatingAdmissionPolicy Let Some Resources Silently Slip Through","As Kyverno's ClusterPolicy heads toward deprecation, teams migrating to ValidatingAdmissionPolicy (CEL) have found that validation silently skipped for certain resource types. Here's the CEL runtime error and failurePolicy trap behind it, and a safe migration path.","2026-10-18",[478,479,524,525,526,527],"validatingadmissionpolicy","kyverno","cel","admission-control",{"path":529,"title":530,"description":531,"date":532,"tags":533},"\u002Fblog\u002Fen\u002Fk3s-vpa-inplaceorrecreate-pod-resize-cgroup","VPA InPlaceOrRecreate Didn't Stop the Restart. Testing InPlacePodVerticalScaling on K3s","Does VPA's InPlaceOrRecreate mode and In-Place Pod Resize really avoid Pod restarts? We test QoS class constraints and the cgroup v1 trap on K3s with real configs and kubectl commands.","2026-10-17",[478,479,534,494,535],"vpa","cgroup",{"path":537,"title":538,"description":539,"date":540,"tags":541},"\u002Fblog\u002Fen\u002Fk3s-pod-security-admission-restricted-servicelb-outage","We Enabled Restricted and ServiceLB Died: The K3s Pod Security Admission Gap No Generic Guide Mentions","Enabling the restricted Pod Security Admission profile on K3s following standard Kubernetes guides can silently kill ServiceLB. Here's why, backed by K3s's own CIS hardening guide, and how to roll it out safely.","2026-10-16",[478,479,542,543,544,545],"pod-security-admission","pod-security-standards","servicelb","security",1792592633347]