[{"data":1,"prerenderedAt":376},["ShallowReactive",2],{"blog-en-kubernetes-cni-overlay-network-troubleshooting-ospf":3,"blog-related-en-kubernetes-cni-overlay-network-troubleshooting-ospf":321,"blog-en-kubernetes-cni-overlay-network-troubleshooting-ospf-alt":310},{"id":4,"title":5,"author":6,"body":7,"date":304,"description":305,"extension":306,"image":307,"locale":308,"meta":309,"navigation":310,"path":311,"seo":312,"stem":313,"tags":314,"__hash__":320},"blog\u002Fblog\u002Fen\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf.md","OSPF Fixes Itself in a Minute — Why Does a Kubernetes CNI Outage Take Half a Day?","Kubo Team",{"type":8,"value":9,"toc":291},"minimark",[10,15,23,31,42,51,55,61,75,100,103,107,113,116,121,146,150,167,174,178,184,187,236,252,255,259,265,268,271,282],[11,12,14],"h2",{"id":13},"ping-works-but-the-pod-doesnt-the-classic-half-broken-failure","Ping Works, But the Pod Doesn't — The Classic \"Half-Broken\" Failure",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf\u002Fsection01.webp",[16,24,25,26,30],{},"The most frustrating failure mode in a Kubernetes overlay network isn't a total outage. ",[27,28,29],"code",{},"ping"," to the pod succeeds, DNS resolution works fine, and yet the application's HTTP requests keep timing out intermittently. This \"half-broken\" state is exactly what wears infrastructure engineers down.",[16,32,33,34,41],{},"In most cases, the root cause is an MTU (Maximum Transmission Unit) mismatch. A pod's network interface inherits its MTU from the CNI plugin, and overlay-style encapsulation eats into that value. According to one troubleshooting guide, the typical MTU is around 1450 for Flannel VXLAN, 1480 for Calico IPIP, 1450 for Calico VXLAN, and 1376 for Weave — all shaved down dozens of bytes from the physical NIC's standard 1500 (see this ",[35,36,40],"a",{"href":37,"rel":38},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-03-20-troubleshoot-mtu-kubernetes\u002Fview",[39],"nofollow","MTU troubleshooting guide","). Small packets like ICMP and DNS pass through unaffected by this reduction, but the moment a TCP session tries to send a larger payload, packets get fragmented and dropped — and the connection appears \"broken.\"",[16,43,44,45,50],{},"This article breaks down why Kubernetes overlay networking gets so complicated in the first place, and walks through a practical triage process you can use in the field. Most of these failures can actually be isolated in minutes once you can see which pod is on which node and how the routes are being resolved. Managed K3s environments like ",[35,46,49],{"href":47,"rel":48},"https:\u002F\u002Fkubo.hexabase.io\u002F",[39],"Kubo",", for instance, are built around exactly this kind of network topology visibility.",[11,52,54],{"id":53},"why-does-ospf-converge-on-its-own-while-kubernetes-cni-leaves-humans-to-do-the-digging","Why Does OSPF Converge on Its Own, While Kubernetes CNI Leaves Humans to Do the Digging?",[16,56,57],{},[19,58],{"alt":59,"src":60},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf\u002Fsection02.webp",[16,62,63,64,68,69,74],{},"OSPF (Open Shortest Path First) is the classic mechanism for handling routing information on traditional physical networks. Each router broadcasts its own connectivity state as an ",[65,66,67],"strong",{},"LSA (Link State Advertisement)",", so every router ends up with an identical topology database. When a topology change occurs — say, a link failure — the affected router floods a new LSA, and every router recalculates the shortest path in parallel. The result is routes that rebuild themselves automatically, with \"a short convergence period and minimal traffic overhead\" (see ",[35,70,73],{"href":71,"rel":72},"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc2328.txt",[39],"RFC 2328: OSPF Version 2",", standardized in 1998).",[16,76,77,78,83,84,87,88,93,94,99],{},"Kubernetes' networking model starts from an entirely different set of assumptions. Kubernetes requires that \"all containers can communicate with each other without NAT\" and that \"a container's IP address is the same from inside and outside\" (see this ",[35,79,82],{"href":80,"rel":81},"https:\u002F\u002Fwww.aquasec.com\u002Fcloud-native-academy\u002Fkubernetes-101\u002Fkubernetes-networking\u002F",[39],"explainer on the Kubernetes networking model's requirements","). But the specification itself doesn't define how to satisfy that requirement — that's left entirely to ",[65,85,86],{},"CNI (Container Network Interface)"," plugins. CNI is a standard that CNCF accepted as an incubating project in 2017; it merely defines how a container runtime (containerd, CRI-O, etc.) invokes a plugin via commands like ADD\u002FDEL when attaching a pod to the network (see ",[35,89,92],{"href":90,"rel":91},"https:\u002F\u002Fwww.cncf.io\u002Fprojects\u002Fcontainer-network-interface-cni\u002F",[39],"CNCF: Container Network Interface (CNI)"," and the ",[35,95,98],{"href":96,"rel":97},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fextend-kubernetes\u002Fcompute-storage-net\u002Fnetwork-plugins\u002F",[39],"official Kubernetes docs on network plugins",").",[16,101,102],{},"In other words, in the OSPF world, fixed pieces of hardware — routers — learn routes from each other. In the Kubernetes world, every time a pod gets scheduled it's assigned a new IP, and the CNI plugin has to reflect that routing information after the fact — an inherently asynchronous process. Unlike OSPF, where the network gear itself converges autonomously, a Kubernetes overlay requires a human to chase down \"which pod is on which node right now, and how is that route being resolved\" using logs and tools. That's the real reason it \"takes half a day.\"",[11,104,106],{"id":105},"flannels-vxlan-vs-calicos-bgp-failures-look-completely-different","Flannel's VXLAN vs. Calico's BGP: Failures Look Completely Different",[16,108,109],{},[19,110],{"alt":111,"src":112},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf\u002Fsection03.webp",[16,114,115],{},"The two most common CNIs in K3s clusters are Flannel and Calico, and they fail in completely different ways — and point you to completely different places to investigate.",[117,118,120],"h3",{"id":119},"flannel-vxlan-the-pitfalls-of-l2-overlay","Flannel VXLAN — The Pitfalls of L2 Overlay",[16,122,123,124,127,128,131,132,137,138,141,142,145],{},"K3s uses Flannel with a VXLAN backend by default, encapsulating packets for transport. Beyond VXLAN, Flannel also offers backends like ",[27,125,126],{},"host-gw"," (routing via node IPs, which requires layer-2 connectivity between all nodes) and ",[27,129,130],{},"wireguard-native"," (encrypted encapsulation) (see ",[35,133,136],{"href":134,"rel":135},"https:\u002F\u002Fdocs.k3s.io\u002Fnetworking\u002Fbasic-network-options",[39],"K3s official docs: Basic Network Options","). One detail worth watching closely is the UDP port: Flannel's VXLAN defaults to UDP port 8472, the Linux kernel's traditional default, while Calico's VXLAN uses UDP port 4789, the IANA standard. These are genuinely different defaults in different implementations, not variations of \"the same port\" — so the first thing to check when a node-to-node firewall might be involved is whether the port matching your actual CNI is blocked. When something breaks, the standard playbook is to check the ",[27,139,140],{},"flanneld"," logs and run ",[27,143,144],{},"ip -d link show flannel.1"," on each node to confirm MTU and encapsulation state.",[117,147,149],{"id":148},"calico-bgp-the-routing-mode-you-choose-matters","Calico BGP — The Routing Mode You Choose Matters",[16,151,152,153,158,159,162,163,166],{},"Calico differs from Flannel in that it supports both a non-overlay BGP mode and overlay modes (IPIP\u002FVXLAN). In BGP mode, Calico shares routes using \"the standard BGP routing protocol,\" peering with the physical network (e.g., ToR switches) so pod IPs can be routed directly even from outside the cluster. Overlay mode, on the other hand, \"works in nearly any network environment,\" at the cost of CPU overhead from encapsulation and a reduced MTU (see ",[35,154,157],{"href":155,"rel":156},"https:\u002F\u002Fdocs.tigera.io\u002Fcalico\u002Flatest\u002Fnetworking\u002Fdetermine-best-networking",[39],"Calico official docs: Determine best networking","). When something fails in BGP mode, you check the BGP peering state of ",[27,160,161],{},"calico-node"," (roughly equivalent to ",[27,164,165],{},"calicoctl node status","); in overlay mode, you're back to suspecting encapsulation and MTU, just like with Flannel. The starting point for investigation is completely different depending on which mode you're running.",[16,168,169,170,173],{},"This fact alone — that the logs and tools you need to check depend entirely on which CNI you're running — is a major reason Kubernetes network incident response ends up depending on a single specialist. Continuing to hand-tune whether you run Flannel or Calico, and BGP or overlay, isn't something you can do as a side project. ",[35,171,49],{"href":47,"rel":172},[39]," offers both of these CNI configurations as selectable options, taking on the cost of that configuration decision itself.",[11,175,177],{"id":176},"the-first-triage-steps-to-run-in-the-field","The First Triage Steps to Run in the Field",[16,179,180],{},[19,181],{"alt":182,"src":183},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf\u002Fsection04.webp",[16,185,186],{},"When you hit a failure where \"ping works but HTTP just hangs,\" the following order — based on experience — tends to minimize guesswork.",[188,189,190,200,213,227],"ol",{},[191,192,193,199],"li",{},[65,194,195,196],{},"Check events with ",[27,197,198],{},"kubectl describe pod",": Obvious failures, like scheduling issues or CNI initialization errors, tend to show up here first",[191,201,202,205,206,208,209,212],{},[65,203,204],{},"Check the CNI DaemonSet logs",": Look at the ",[27,207,161],{}," or ",[27,210,211],{},"kube-flannel"," pod logs for clues like IP pool exhaustion or a broken BGP peering session",[191,214,215,218,219,222,223,226],{},[65,216,217],{},"Check the MTU inside the pod",": Run ",[27,220,221],{},"ip link show eth0"," to see the actual MTU value, and compare it against the recommended value for your CNI (Flannel VXLAN = 1450, Calico IPIP = 1480, etc.) (see this ",[35,224,40],{"href":37,"rel":225},[39],")",[191,228,229,235],{},[65,230,231,232],{},"Check encapsulated packets with ",[27,233,234],{},"tcpdump",": Visually confirm on the node-to-node interface that VXLAN or BGP packets are actually flowing as expected",[16,237,238,239,242,243,246,247,99],{},"Beyond that, there are a few well-known environment-specific pitfalls worth checking: whether IP forwarding has been disabled (",[27,240,241],{},"sysctl net.ipv4.ip_forward","), whether the bridge netfilter setting has been disabled (",[27,244,245],{},"sysctl net.bridge.bridge-nf-call-iptables","), and — on some clouds like AWS — whether source\u002Fdestination checks are blocking overlay traffic (see this ",[35,248,251],{"href":249,"rel":250},"https:\u002F\u002Fgoteleport.com\u002Fblog\u002Ftroubleshooting-kubernetes-networking\u002F",[39],"Kubernetes network troubleshooting guide",[16,253,254],{},"None of these steps \"just fix themselves\" the way OSPF does. A human has to trace the logs, use the tools, and eliminate causes one by one. Whether or not you've turned this tedious process into a repeatable playbook is what separates a one-hour incident from a half-day one.",[11,256,258],{"id":257},"summary","Summary",[16,260,261],{},[19,262],{"alt":263,"src":264},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf\u002Fsection05.webp",[16,266,267],{},"The fact that Kubernetes overlay networking doesn't self-converge the way OSPF does isn't negligence or a design flaw. It's a consequence of the CNI's inherent asynchrony: pods are dynamically created and destroyed, and routing information has to be reflected after the fact, every single time. It's also worth remembering that the choice between Flannel's VXLAN and Calico's BGP significantly shapes where you should look first when something breaks.",[16,269,270],{},"This \"standardized triage process\" is the real cost of operations, and if only one person on the team knows the steps, you're carrying the risk of that incident response staying dependent on a single individual indefinitely. In practice, the cost of running this triage process from scratch every single time is one of the most invisible sources of technical debt in running your own K3s cluster.",[16,272,273,276,277,281],{},[35,274,49],{"href":47,"rel":275},[39]," is a managed Kubernetes service built on K3s that lets you choose either a Flannel or Calico configuration, while visualizing network topology and pod placement through the Captain UI. Instead of chasing routing tables and CNI logs by hand, it's designed to let you see how routes are actually being resolved, right on screen. Rather than continuing to hand-tune EKS or AKS yourself, ",[35,278,280],{"href":47,"rel":279},[39],"Kubo Cloud"," can take on that operational burden starting at ¥48,000 per month.",[16,283,284,285,290],{},"If you feel like you're burning time on CNI troubleshooting every time it comes up, ",[35,286,289],{"href":287,"rel":288},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[39],"get in touch"," and talk through your current setup. You can get the same pure Kubernetes experience at roughly 58% of the cost of EKS, with less operational overhead.",{"title":292,"searchDepth":293,"depth":293,"links":294},"",2,[295,296,297,302,303],{"id":13,"depth":293,"text":14},{"id":53,"depth":293,"text":54},{"id":105,"depth":293,"text":106,"children":298},[299,301],{"id":119,"depth":300,"text":120},3,{"id":148,"depth":300,"text":149},{"id":176,"depth":293,"text":177},{"id":257,"depth":293,"text":258},"2026-08-16","Why does troubleshooting a Kubernetes overlay network take so long? This article breaks down how CNI works, the differences between Flannel VXLAN and Calico BGP, and the step-by-step process for isolating MTU mismatches that every K3s operator should know.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf\u002Feyecatch.webp","en",{},true,"\u002Fblog\u002Fen\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf",{"title":5,"description":305},"blog\u002Fen\u002Fkubernetes-cni-overlay-network-troubleshooting-ospf",[315,316,317,318,319],"k3s","kubernetes","cni","overlay-network","networking","j7D8W4DtmQFDG3T6oaxNF1Pn9WQs2a4SdQlPaKh0WeY",[322,330,338,349,359,367],{"path":323,"title":324,"description":325,"date":326,"tags":327},"\u002Fblog\u002Fen\u002Fkubernetes-networking-cni-service-mesh-network-engineer","You Can Calculate Subnets, But Your Pods Won't Connect: The First Wall in Kubernetes Networking","You're great at subnet design, so why does Pod-to-Pod communication in Kubernetes trip you up every time? We break down the fundamentals of Kubernetes networking—CNI, Service Mesh, and NetworkPolicy—by contrasting them with traditional network design.","2026-07-22",[316,315,319,317,328,329],"service-mesh","network-policy",{"path":331,"title":332,"description":333,"date":334,"tags":335},"\u002Fblog\u002Fen\u002Fkubevirt-calico-live-migration-networking","Moving a VM Doesn't Have to Break the Connection: Inside KubeVirt and Calico's Live Migration Magic","Why doesn't live migrating a VM (KubeVirt) between Kubernetes nodes break the connection? We break down Calico's IP persistence and BGP route convergence, and what it means for teams moving off VMware.","2026-08-07",[315,316,336,319,337],"kubevirt","managed-kubernetes",{"path":339,"title":340,"description":341,"date":342,"tags":343},"\u002Fblog\u002Fen\u002Fcilium-ebpf-kubernetes-networking","Revolutionizing Kubernetes Networking with Cilium and eBPF","Discover how Cilium uses eBPF to transform Kubernetes networking. Compare with traditional CNIs, explore L7 policies, and leverage Hubble for observability.","2026-05-27",[344,345,316,319,317,346,347,348],"cilium","ebpf","cncf","security","hubble",{"path":350,"title":351,"description":352,"date":353,"tags":354},"\u002Fblog\u002Fen\u002Fk3s-container-image-security-supply-chain-checklist","'Scanned' Is Not a Production Clearance Certificate: K3s Container Image Security From Signing to Admission Control","Container security guidelines often stop at 'we run a scanner.' This guide walks through the practical checklist you need to pass before production in K3s: minimal base images, vulnerability scanning, SBOM generation, signing, and admission control.","2026-08-24",[315,316,355,356,357,358],"container-security","image-scanning","sbom","supply-chain-security",{"path":360,"title":361,"description":362,"date":363,"tags":364},"\u002Fblog\u002Fen\u002Fk3s-harbor-private-registry-docker-hub-rate-limit","The Morning Docker Hub's Free Tier Freezes Up, Your K3s Cluster Quietly Grinds to a Halt: When to Run Your Own Harbor","Docker Hub's pull rate limit is an increasingly real risk to image pulls on production K3s clusters. Kubo explains the design decisions and hidden costs of self-hosting Harbor, a CNCF Graduated project.","2026-08-23",[315,316,365,366,346],"harbor","container-registry",{"path":368,"title":369,"description":370,"date":371,"tags":372},"\u002Fblog\u002Fen\u002Fkubernetes-cost-management-eks-aks-billing-visibility","Your EKS Bill Only Makes Sense at Month-End: Why Kubernetes Costs Are Structurally 'Discovered Too Late'","Why do Kubernetes costs on EKS\u002FAKS balloon unexpectedly? We break down how autoscaling and cross-AZ billing hide costs, and explore how K3s-based managed infrastructure turns them into a fixed cost.","2026-08-22",[315,316,373,337,374,375],"cost-optimization","aks","finops",1787649516797]