[{"data":1,"prerenderedAt":576},["ShallowReactive",2],{"blog-en-k3s-baremetal-loadbalancer-bgp-metallb-cilium":3,"blog-related-en-k3s-baremetal-loadbalancer-bgp-metallb-cilium":524,"blog-en-k3s-baremetal-loadbalancer-bgp-metallb-cilium-alt":513},{"id":4,"title":5,"author":6,"body":7,"date":508,"description":509,"extension":510,"image":193,"locale":511,"meta":512,"navigation":513,"path":514,"seo":515,"stem":516,"tags":517,"__hash__":523},"blog\u002Fblog\u002Fen\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium.md","Bare-Metal K3s LoadBalancer: When ARP Hits Its Limit, Design for BGP","Kubo Team",{"type":8,"value":9,"toc":497},"minimark",[10,24,31,36,50,63,80,86,95,99,108,111,127,136,142,146,155,168,187,384,391,400,406,415,419,422,427,442,446,449,453,460,466,470,473,476,493],[11,12,13,14,18,19,23],"p",{},"Factory floors, data center racks, the back room of a retail store — as edge Kubernetes deployments spread into these environments, one problem always surfaces sooner or later: ",[15,16,17],"strong",{},"designing a K3s load balancer around BGP",", a concern you never had to think about in the cloud. On EKS or AKS, writing ",[20,21,22],"code",{},"type: LoadBalancer"," is enough to get an external IP automatically. But on bare metal, where no cloud API exists, you have to build that mechanism yourself.",[11,25,26],{},[27,28],"img",{"alt":29,"src":30},"Architecture comparison: three patterns for how traffic enters a cluster — cloud LB, L2 mode, and BGP mode","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium\u002Fsection01.webp",[32,33,35],"h2",{"id":34},"how-does-service-type-loadbalancer-work-without-a-cloud-lb","How Does \"Service type: LoadBalancer\" Work Without a Cloud LB?",[11,37,38,39,46,47,49],{},"According to the ",[40,41,45],"a",{"href":42,"rel":43},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fservices-networking\u002Fservice\u002F#loadbalancer",[44],"nofollow","Kubernetes documentation",", a ",[20,48,22],{}," Service normally goes through the Cloud Controller Manager, which calls the cloud provider's load balancer API to automatically assign an external IP. This mechanism is tightly coupled to public clouds like AWS, Azure, and GCP.",[11,51,52,53,56,57,62],{},"In bare-metal environments — factories, branch sites, edge locations — ",[15,54,55],{},"there is no cloud API",", so this automatic assignment simply doesn't happen. This is where software load balancers like MetalLB come in: they use standard routing protocols to \"announce\" an external IP onto the network. The ",[40,58,61],{"href":59,"rel":60},"https:\u002F\u002Fmetallb.universe.tf\u002Fconcepts\u002F",[44],"MetalLB documentation"," describes two operating modes: Layer2 (ARP\u002FNDP) and BGP.",[11,64,65,66,69,70,75,76,79],{},"If you're running K3s, there's one more thing to watch for. K3s ships with a built-in load balancer called ",[20,67,68],{},"ServiceLB"," (formerly klipper-lb), enabled by default. If you want to install MetalLB or another implementation instead, the ",[40,71,74],{"href":72,"rel":73},"https:\u002F\u002Fdocs.k3s.io\u002Fnetworking\u002Fnetworking-services",[44],"K3s documentation"," requires you to pass ",[20,77,78],{},"--disable servicelb"," on every server node — otherwise the two implementations will fight over which one owns the external IP assignment.",[11,81,82],{},[27,83],{"alt":84,"src":85},"Flowchart: three steps to disable ServiceLB before deploying MetalLB in a K3s cluster","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium\u002Fsection02.webp",[11,87,88,89,94],{},"If you'd rather not own this design decision from scratch, a K3s-based managed service like ",[40,90,93],{"href":91,"rel":92},"https:\u002F\u002Fkubo.hexabase.io\u002F",[44],"Kubo"," is another option — it removes the need to build the network design yourself.",[32,96,98],{"id":97},"where-layer2-mode-hits-a-wall-a-single-node-arp-caches-and-bandwidth","Where Layer2 Mode Hits a Wall: A Single Node, ARP Caches, and Bandwidth",[11,100,101,102,107],{},"MetalLB's Layer2 mode, according to its ",[40,103,106],{"href":104,"rel":105},"https:\u002F\u002Fmetallb.universe.tf\u002Fconcepts\u002Flayer2\u002F",[44],"official documentation",", works through leader election: \"one node takes responsibility for announcing a service IP to the local network.\" ARP (IPv4) or NDP (IPv6) responses then make it look as if a single machine holds multiple external IP addresses.",[11,109,110],{},"That's fine in a test environment. But in production on-prem or edge deployments, two structural limits appear:",[112,113,114,121],"ul",{},[115,116,117,120],"li",{},[15,118,119],{},"Single-node bottleneck",": All traffic to a given external IP funnels into whichever node was elected leader. No matter how many nodes you add to the cluster, that one service's inbound bandwidth can never exceed the physical bandwidth of a single node.",[115,122,123,126],{},[15,124,125],{},"Failover delay",": When the leader node goes down, a new node has to resend ARP responses to tell clients the target has switched. In most environments this converges within seconds, but clients holding stale ARP cache entries can experience delays measured in minutes.",[11,128,129,130,135],{},"The ",[40,131,134],{"href":132,"rel":133},"https:\u002F\u002Fwww.redhat.com\u002Fen\u002Fblog\u002Fmetallb-in-bgp-mode",[44],"Red Hat engineering blog"," makes the same point explicitly: Layer2 mode turns a node into \"a single point of failure that limits performance.\" As a K3s cluster grows from three nodes to ten in an edge rollout, this constraint becomes a hard ceiling on that service's scalability.",[11,137,138],{},[27,139],{"alt":140,"src":141},"Architecture diagram: in L2 mode, traffic to one external IP concentrates on a single leader node, which caps overall service bandwidth","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium\u002Fsection03.webp",[32,143,145],{"id":144},"moving-to-bgp-metallbs-bgp-mode-vs-ciliums-bgp-control-plane","Moving to BGP: MetalLB's BGP Mode vs. Cilium's BGP Control Plane",[11,147,148,149,154],{},"The standard way past Layer2's limits is BGP mode. According to ",[40,150,153],{"href":151,"rel":152},"https:\u002F\u002Fmetallb.universe.tf\u002Fconcepts\u002Fbgp\u002F",[44],"MetalLB's BGP concept guide",", in BGP mode every node in the cluster establishes a BGP peering session with the upstream router, and the same external IP is advertised simultaneously from multiple nodes. If the router supports ECMP (Equal-Cost Multi-Path), traffic is actually load-balanced across those nodes based on packet hashing.",[11,156,129,157,161,162,167],{},[40,158,160],{"href":132,"rel":159},[44],"Red Hat blog"," sums up BGP mode's advantages as stateless load balancing, simultaneous advertisement from multiple nodes, and fast failure detection via the BFD protocol. MetalLB supports delegating the actual BGP session handling to an FRR-based backend called ",[40,163,166],{"href":164,"rel":165},"https:\u002F\u002Fwww.redhat.com\u002Fen\u002Fblog\u002Ffrr-k8s-bgp-backend-metallb",[44],"FRR-K8s",", which unlocks capabilities Layer2 mode doesn't have, such as combining BFD sessions with IPv6 advertisement.",[11,169,170,171,174,175,180,181,186],{},"There's a second option if your CNI is Cilium: ",[15,172,173],{},"Cilium's BGP Control Plane",". According to the ",[40,176,179],{"href":177,"rel":178},"https:\u002F\u002Fdocs.cilium.io\u002Fen\u002Fstable\u002Fnetwork\u002Fbgp-control-plane\u002Fbgp-control-plane\u002F",[44],"Cilium documentation",", the BGP Control Plane advertises Pod network and Service external IPs over BGP, while ",[40,182,185],{"href":183,"rel":184},"https:\u002F\u002Fdocs.cilium.io\u002Fen\u002Fstable\u002Fnetwork\u002Flb-ipam\u002F",[44],"LB-IPAM"," handles IP address allocation. If your cluster already runs Cilium as its CNI, you can consolidate BGP load balancing entirely into the CNI layer without adding MetalLB as a separate component.",[188,189,194],"pre",{"className":190,"code":191,"language":192,"meta":193,"style":193},"language-yaml shiki shiki-themes tokyo-night","# Example: simplified Cilium BGP Control Plane configuration\napiVersion: cilium.io\u002Fv2alpha1\nkind: CiliumBGPPeeringPolicy\nmetadata:\n  name: rack01-bgp-policy\nspec:\n  nodeSelector:\n    matchLabels:\n      rack: \"01\"\n  virtualRouters:\n    - localASN: 65001\n      neighbors:\n        - peerAddress: \"10.0.0.1\u002F32\"\n          peerASN: 65000\n      serviceSelector:\n        matchLabels:\n          advertise: \"bgp\"\n","yaml","",[20,195,196,205,220,231,240,251,259,267,275,292,300,316,324,342,353,361,369],{"__ignoreMap":193},[197,198,201],"span",{"class":199,"line":200},"line",1,[197,202,204],{"class":203},"sbD-w","# Example: simplified Cilium BGP Control Plane configuration\n",[197,206,208,212,216],{"class":199,"line":207},2,[197,209,211],{"class":210},"s0U2E","apiVersion",[197,213,215],{"class":214},"sAklC",":",[197,217,219],{"class":218},"sPY7s"," cilium.io\u002Fv2alpha1\n",[197,221,223,226,228],{"class":199,"line":222},3,[197,224,225],{"class":210},"kind",[197,227,215],{"class":214},[197,229,230],{"class":218}," CiliumBGPPeeringPolicy\n",[197,232,234,237],{"class":199,"line":233},4,[197,235,236],{"class":210},"metadata",[197,238,239],{"class":214},":\n",[197,241,243,246,248],{"class":199,"line":242},5,[197,244,245],{"class":210},"  name",[197,247,215],{"class":214},[197,249,250],{"class":218}," rack01-bgp-policy\n",[197,252,254,257],{"class":199,"line":253},6,[197,255,256],{"class":210},"spec",[197,258,239],{"class":214},[197,260,262,265],{"class":199,"line":261},7,[197,263,264],{"class":210},"  nodeSelector",[197,266,239],{"class":214},[197,268,270,273],{"class":199,"line":269},8,[197,271,272],{"class":210},"    matchLabels",[197,274,239],{"class":214},[197,276,278,281,283,286,289],{"class":199,"line":277},9,[197,279,280],{"class":210},"      rack",[197,282,215],{"class":214},[197,284,285],{"class":214}," \"",[197,287,288],{"class":218},"01",[197,290,291],{"class":214},"\"\n",[197,293,295,298],{"class":199,"line":294},10,[197,296,297],{"class":210},"  virtualRouters",[197,299,239],{"class":214},[197,301,303,307,310,312],{"class":199,"line":302},11,[197,304,306],{"class":305},"sgJMe","    -",[197,308,309],{"class":210}," localASN",[197,311,215],{"class":214},[197,313,315],{"class":314},"sOJ5S"," 65001\n",[197,317,319,322],{"class":199,"line":318},12,[197,320,321],{"class":210},"      neighbors",[197,323,239],{"class":214},[197,325,327,330,333,335,337,340],{"class":199,"line":326},13,[197,328,329],{"class":305},"        -",[197,331,332],{"class":210}," peerAddress",[197,334,215],{"class":214},[197,336,285],{"class":214},[197,338,339],{"class":218},"10.0.0.1\u002F32",[197,341,291],{"class":214},[197,343,345,348,350],{"class":199,"line":344},14,[197,346,347],{"class":210},"          peerASN",[197,349,215],{"class":214},[197,351,352],{"class":314}," 65000\n",[197,354,356,359],{"class":199,"line":355},15,[197,357,358],{"class":210},"      serviceSelector",[197,360,239],{"class":214},[197,362,364,367],{"class":199,"line":363},16,[197,365,366],{"class":210},"        matchLabels",[197,368,239],{"class":214},[197,370,372,375,377,379,382],{"class":199,"line":371},17,[197,373,374],{"class":210},"          advertise",[197,376,215],{"class":214},[197,378,285],{"class":214},[197,380,381],{"class":218},"bgp",[197,383,291],{"class":214},[11,385,386,387,390],{},"Note that MetalLB itself used to offer an integrated mode called ",[20,388,389],{},"MetalLB BGP ControlPlane"," that paired with Cilium's data plane, but that mode is now in a maintenance-only phase receiving security fixes, with migration toward Cilium's newer BGP Control Plane recommended instead. Which path makes sense depends heavily on your existing CNI choice — that's the real starting point for a migration decision.",[11,392,393,394,399],{},"If you want BGP handled entirely by dedicated switches, ",[40,395,398],{"href":396,"rel":397},"https:\u002F\u002Fwww.infoq.com\u002Fjp\u002Fnews\u002F2020\u002F08\u002Fporter-kubernetes-bare-metal\u002F",[44],"alternative OSS implementations like Porter"," exist too, but in terms of track record and documentation depth within K3s environments, MetalLB and Cilium remain the two realistic starting points.",[11,401,402],{},[27,403],{"alt":404,"src":405},"Comparison diagram: MetalLB BGP mode vs. Cilium BGP Control Plane, showing ECMP-distributed traffic","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium\u002Fsection04.webp",[11,407,408,409,414],{},"For teams that would rather not own this migration end-to-end, ",[40,410,413],{"href":411,"rel":412},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[44],"Kubo On-Premise"," is an option. It's K3s-based, supports air-gapped environments, and lets an operations team hand off network design — including BGP peering — without vendor lock-in. The tighter your external network connectivity requirements (think financial institutions or manufacturing edge sites), the bigger the payoff.",[32,416,418],{"id":417},"three-things-to-check-before-you-migrate","Three Things to Check Before You Migrate",[11,420,421],{},"Moving from Layer2 mode to BGP mode isn't something you can finish purely by configuring the Kubernetes cluster. In practice, there are three things you always need to confirm.",[423,424,426],"h3",{"id":425},"_1-have-you-disabled-k3ss-built-in-servicelb-klipper","1. Have you disabled K3s's built-in ServiceLB (klipper)?",[11,428,429,430,432,433,438,439,441],{},"As mentioned above, K3s enables ",[20,431,68],{}," by default. Per ",[40,434,437],{"href":435,"rel":436},"https:\u002F\u002Fmetallb.universe.tf\u002Fconfiguration\u002Fk3s\u002F",[44],"MetalLB's K3s configuration guide",", installing MetalLB without disabling it first causes the two implementations to compete over external IP assignment, leading to unpredictable load balancer behavior. Set ",[20,440,78],{}," on every server node first, then install MetalLB or enable Cilium's BGP features — the order matters.",[423,443,445],{"id":444},"_2-do-your-upstream-routersswitches-support-bgp-and-have-you-aligned-with-the-network-team","2. Do your upstream routers\u002Fswitches support BGP, and have you aligned with the network team?",[11,447,448],{},"BGP configuration doesn't stop at the cluster boundary. You also need BGP peer settings (ASN, peer IP addresses, route filters) on the upstream router or switch side, and in most organizations that falls under the network team's authority. This alignment process — unnecessary under Layer2 mode — tends to be the biggest practical hurdle in a BGP migration.",[423,450,452],{"id":451},"_3-have-you-verified-failover-behavior","3. Have you verified failover behavior?",[11,454,455,456,459],{},"Whether ECMP routes get withdrawn correctly on node failure, and whether BFD-based fast detection actually works, are things you must verify before going to production. As the ",[40,457,160],{"href":132,"rel":458},[44]," notes, combining BFD can shorten failure detection to well under a second — but that depends entirely on router-side support, which is why pre-verification is non-negotiable.",[11,461,462],{},[27,463],{"alt":464,"src":465},"Process diagram: disable ServiceLB, align with the network team, then verify failover — three steps","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium\u002Fsection05.webp",[32,467,469],{"id":468},"summary","Summary",[11,471,472],{},"For bare-metal and edge K3s clusters that can't rely on a cloud load balancer, Layer2 (ARP) mode load balancing is really only a compromise fit for test environments. It inevitably runs into structural limits — traffic concentrating on a single node, delayed failover — which is why migrating to BGP mode becomes practically inevitable for production use.",[11,474,475],{},"Which path to take depends on your CNI setup. If you're already running Cilium, consolidating into its BGP Control Plane and LB-IPAM is the sensible choice; for any other CNI, MetalLB's BGP mode is the realistic option. Either way, the deciding factor in a successful migration is doing the unglamorous groundwork: disabling K3s's ServiceLB and reaching agreement with the network team.",[11,477,478,479,482,483,486,487,492],{},"As Kubernetes spreads beyond the cloud into factories, retail stores, and even vehicles, this kind of network design decision becomes unavoidable. K3s-based managed services like ",[40,480,93],{"href":91,"rel":481},[44]," and ",[40,484,413],{"href":411,"rel":485},[44]," can take on this design and operational burden for you. If you'd rather not shoulder the verification cost of a BGP migration yourselves, ",[40,488,491],{"href":489,"rel":490},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[44],"reaching out"," is worth considering.",[494,495,496],"style",{},"html pre.shiki code .sbD-w, html code.shiki .sbD-w{--shiki-default:#51597D;--shiki-default-font-style:italic}html pre.shiki code .s0U2E, html code.shiki .s0U2E{--shiki-default:#F7768E}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html pre.shiki code .sgJMe, html code.shiki .sgJMe{--shiki-default:#9ABDF5}html pre.shiki code .sOJ5S, html code.shiki .sOJ5S{--shiki-default:#FF9E64}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":193,"searchDepth":207,"depth":207,"links":498},[499,500,501,502,507],{"id":34,"depth":207,"text":35},{"id":97,"depth":207,"text":98},{"id":144,"depth":207,"text":145},{"id":417,"depth":207,"text":418,"children":503},[504,505,506],{"id":425,"depth":222,"text":426},{"id":444,"depth":222,"text":445},{"id":451,"depth":222,"text":452},{"id":468,"depth":207,"text":469},"2026-08-26","When you can't rely on a cloud load balancer for on-prem or edge K3s clusters, why does Layer2 (ARP) load balancing eventually hit a wall? A practical look at MetalLB's BGP mode vs. Cilium's BGP Control Plane, and how to decide when to migrate.","md","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium",{"title":5,"description":509},"blog\u002Fen\u002Fk3s-baremetal-loadbalancer-bgp-metallb-cilium",[518,519,520,381,521,522],"k3s","kubernetes","metallb","load-balancer","edge-computing","2yHIPw14McTqGGS6gBIi_ePxJ6hbD7Jgd0tf-p1L6MQ",[525,533,541,550,558,567],{"path":526,"title":527,"description":528,"date":529,"tags":530},"\u002Fblog\u002Fen\u002Fk3s-prometheus-agent-mode-stale-marker-edge-monitoring","We Switched K3s Edge Prometheus to Agent Mode. The Moment a Target Disappeared, So Did Its Metrics","Running Prometheus Agent mode on K3s edge clusters can leave stale values in your central dashboard when a target disappears, depending on how stale markers are handled. Here's how to design K3s edge monitoring correctly.","2026-11-03",[518,519,531,522,532],"prometheus","observability",{"path":534,"title":535,"description":536,"date":537,"tags":538},"\u002Fblog\u002Fen\u002Fk3s-system-upgrade-controller-plan-stuck-node-drain","The Plan Never Disappeared. Why K3s's system-upgrade-controller Got Stuck on One Node","We set up automated upgrades with K3s's system-upgrade-controller, and the Plan stopped progressing on a single node. The culprits were PodDisruptionBudgets and taints. Here's how the mechanism works, how to recover, and how Kubo helps.","2026-10-31",[518,519,539,540,522],"system-upgrade-controller","rolling-upgrade",{"path":542,"title":543,"description":544,"date":545,"tags":546},"\u002Fblog\u002Fen\u002Fk3s-spegel-embedded-registry-mirror-airgap-node-join","K3s's Embedded Registry Mirror Was Supposed to Just Work. Then a New Edge Node Couldn't Pull Images","K3s's embedded registry mirror (Spegel) is supposed to work cluster-wide, but a newly added edge node alone failed to pull images. The cause: an undistributed registries.yaml, credential drift, and a known containerd 2.0 bug. Here's the correct distribution workflow for air-gapped operations.","2026-10-30",[518,519,547,548,549,522],"spegel","registry-mirror","airgap",{"path":551,"title":552,"description":553,"date":554,"tags":555},"\u002Fblog\u002Fen\u002Fk3s-microshift-etcd-ostree-edge-comparison","Is Pinning etcd at the Edge a Trap or a Trade-off? K3s vs MicroShift and the Real Cost of OSTree Auto-Rollback","K3s and MicroShift are both called \"lightweight Kubernetes,\" but their assumptions about datastores and OS update models are fundamentally different. We compare etcd lock-in and OSTree auto-rollback to help you choose for production edge deployments.","2026-10-29",[518,556,519,522,557],"microshift","redhat",{"path":559,"title":560,"description":561,"date":562,"tags":563},"\u002Fblog\u002Fen\u002Fk3s-vs-microk8s-ha-etcd-dqlite-edge-production","K3s vs MicroK8s: Which One Actually Wins in Production? The HA Fork Between dqlite and etcd","K3s vs MicroK8s: both are called 'lightweight Kubernetes,' but the real production divide is HA architecture. We compare K3s's embedded etcd and MicroK8s's dqlite using official documentation data.","2026-10-25",[518,564,519,565,522,566],"microk8s","high-availability","comparison",{"path":568,"title":569,"description":570,"date":571,"tags":572},"\u002Fblog\u002Fen\u002Fk3s-traefik-servicelb-ingress-nginx-port-conflict","You Disabled Traefik on K3s, But ingress-nginx Still Can't Grab Port 443. Meet ServiceLB, the Roommate You Forgot About","You ran the K3s disable-traefik flag but ingress-nginx still can't bind port 443. The culprit is ServiceLB (klipper-lb), another default K3s component. Here's how to diagnose it, disable it correctly, and what changes if you're running MetalLB.","2026-09-19",[518,519,573,574,575,521],"traefik","servicelb","ingress-nginx",1794902494231]