Ping Works, But the Pod Doesn't — The Classic "Half-Broken" Failure
The most frustrating failure mode in a Kubernetes overlay network isn't a total outage. ping to the pod succeeds, DNS resolution works fine, and yet the application's HTTP requests keep timing out intermittently. This "half-broken" state is exactly what wears infrastructure engineers down.
In most cases, the root cause is an MTU (Maximum Transmission Unit) mismatch. A pod's network interface inherits its MTU from the CNI plugin, and overlay-style encapsulation eats into that value. According to one troubleshooting guide, the typical MTU is around 1450 for Flannel VXLAN, 1480 for Calico IPIP, 1450 for Calico VXLAN, and 1376 for Weave — all shaved down dozens of bytes from the physical NIC's standard 1500 (see this MTU troubleshooting guide). Small packets like ICMP and DNS pass through unaffected by this reduction, but the moment a TCP session tries to send a larger payload, packets get fragmented and dropped — and the connection appears "broken."
This article breaks down why Kubernetes overlay networking gets so complicated in the first place, and walks through a practical triage process you can use in the field. Most of these failures can actually be isolated in minutes once you can see which pod is on which node and how the routes are being resolved. Managed K3s environments like Kubo, for instance, are built around exactly this kind of network topology visibility.
Why Does OSPF Converge on Its Own, While Kubernetes CNI Leaves Humans to Do the Digging?
OSPF (Open Shortest Path First) is the classic mechanism for handling routing information on traditional physical networks. Each router broadcasts its own connectivity state as an LSA (Link State Advertisement), so every router ends up with an identical topology database. When a topology change occurs — say, a link failure — the affected router floods a new LSA, and every router recalculates the shortest path in parallel. The result is routes that rebuild themselves automatically, with "a short convergence period and minimal traffic overhead" (see RFC 2328: OSPF Version 2, standardized in 1998).
Kubernetes' networking model starts from an entirely different set of assumptions. Kubernetes requires that "all containers can communicate with each other without NAT" and that "a container's IP address is the same from inside and outside" (see this explainer on the Kubernetes networking model's requirements). But the specification itself doesn't define how to satisfy that requirement — that's left entirely to CNI (Container Network Interface) plugins. CNI is a standard that CNCF accepted as an incubating project in 2017; it merely defines how a container runtime (containerd, CRI-O, etc.) invokes a plugin via commands like ADD/DEL when attaching a pod to the network (see CNCF: Container Network Interface (CNI) and the official Kubernetes docs on network plugins).
In other words, in the OSPF world, fixed pieces of hardware — routers — learn routes from each other. In the Kubernetes world, every time a pod gets scheduled it's assigned a new IP, and the CNI plugin has to reflect that routing information after the fact — an inherently asynchronous process. Unlike OSPF, where the network gear itself converges autonomously, a Kubernetes overlay requires a human to chase down "which pod is on which node right now, and how is that route being resolved" using logs and tools. That's the real reason it "takes half a day."
Flannel's VXLAN vs. Calico's BGP: Failures Look Completely Different
The two most common CNIs in K3s clusters are Flannel and Calico, and they fail in completely different ways — and point you to completely different places to investigate.
Flannel VXLAN — The Pitfalls of L2 Overlay
K3s uses Flannel with a VXLAN backend by default, encapsulating packets for transport. Beyond VXLAN, Flannel also offers backends like host-gw (routing via node IPs, which requires layer-2 connectivity between all nodes) and wireguard-native (encrypted encapsulation) (see K3s official docs: Basic Network Options). One detail worth watching closely is the UDP port: Flannel's VXLAN defaults to UDP port 8472, the Linux kernel's traditional default, while Calico's VXLAN uses UDP port 4789, the IANA standard. These are genuinely different defaults in different implementations, not variations of "the same port" — so the first thing to check when a node-to-node firewall might be involved is whether the port matching your actual CNI is blocked. When something breaks, the standard playbook is to check the flanneld logs and run ip -d link show flannel.1 on each node to confirm MTU and encapsulation state.
Calico BGP — The Routing Mode You Choose Matters
Calico differs from Flannel in that it supports both a non-overlay BGP mode and overlay modes (IPIP/VXLAN). In BGP mode, Calico shares routes using "the standard BGP routing protocol," peering with the physical network (e.g., ToR switches) so pod IPs can be routed directly even from outside the cluster. Overlay mode, on the other hand, "works in nearly any network environment," at the cost of CPU overhead from encapsulation and a reduced MTU (see Calico official docs: Determine best networking). When something fails in BGP mode, you check the BGP peering state of calico-node (roughly equivalent to calicoctl node status); in overlay mode, you're back to suspecting encapsulation and MTU, just like with Flannel. The starting point for investigation is completely different depending on which mode you're running.
This fact alone — that the logs and tools you need to check depend entirely on which CNI you're running — is a major reason Kubernetes network incident response ends up depending on a single specialist. Continuing to hand-tune whether you run Flannel or Calico, and BGP or overlay, isn't something you can do as a side project. Kubo offers both of these CNI configurations as selectable options, taking on the cost of that configuration decision itself.
The First Triage Steps to Run in the Field
When you hit a failure where "ping works but HTTP just hangs," the following order — based on experience — tends to minimize guesswork.
- Check events with
kubectl describe pod: Obvious failures, like scheduling issues or CNI initialization errors, tend to show up here first - Check the CNI DaemonSet logs: Look at the
calico-nodeorkube-flannelpod logs for clues like IP pool exhaustion or a broken BGP peering session - Check the MTU inside the pod: Run
ip link show eth0to see the actual MTU value, and compare it against the recommended value for your CNI (Flannel VXLAN = 1450, Calico IPIP = 1480, etc.) (see this MTU troubleshooting guide) - Check encapsulated packets with
tcpdump: Visually confirm on the node-to-node interface that VXLAN or BGP packets are actually flowing as expected
Beyond that, there are a few well-known environment-specific pitfalls worth checking: whether IP forwarding has been disabled (sysctl net.ipv4.ip_forward), whether the bridge netfilter setting has been disabled (sysctl net.bridge.bridge-nf-call-iptables), and — on some clouds like AWS — whether source/destination checks are blocking overlay traffic (see this Kubernetes network troubleshooting guide).
None of these steps "just fix themselves" the way OSPF does. A human has to trace the logs, use the tools, and eliminate causes one by one. Whether or not you've turned this tedious process into a repeatable playbook is what separates a one-hour incident from a half-day one.
Summary
The fact that Kubernetes overlay networking doesn't self-converge the way OSPF does isn't negligence or a design flaw. It's a consequence of the CNI's inherent asynchrony: pods are dynamically created and destroyed, and routing information has to be reflected after the fact, every single time. It's also worth remembering that the choice between Flannel's VXLAN and Calico's BGP significantly shapes where you should look first when something breaks.
This "standardized triage process" is the real cost of operations, and if only one person on the team knows the steps, you're carrying the risk of that incident response staying dependent on a single individual indefinitely. In practice, the cost of running this triage process from scratch every single time is one of the most invisible sources of technical debt in running your own K3s cluster.
Kubo is a managed Kubernetes service built on K3s that lets you choose either a Flannel or Calico configuration, while visualizing network topology and pod placement through the Captain UI. Instead of chasing routing tables and CNI logs by hand, it's designed to let you see how routes are actually being resolved, right on screen. Rather than continuing to hand-tune EKS or AKS yourself, Kubo Cloud can take on that operational burden starting at ¥48,000 per month.
If you feel like you're burning time on CNI troubleshooting every time it comes up, get in touch and talk through your current setup. You can get the same pure Kubernetes experience at roughly 58% of the cost of EKS, with less operational overhead.