[{"data":1,"prerenderedAt":342},["ShallowReactive",2],{"blog-en-k3s-cert-manager-servicelb-http01-challenge-renewal-failure":3,"blog-related-en-k3s-cert-manager-servicelb-http01-challenge-renewal-failure":293,"blog-en-k3s-cert-manager-servicelb-http01-challenge-renewal-failure-alt":281},{"id":4,"title":5,"author":6,"body":7,"date":275,"description":276,"extension":277,"image":278,"locale":279,"meta":280,"navigation":281,"path":282,"seo":283,"stem":284,"tags":285,"__hash__":292},"blog\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure.md","K3s Wouldn't Renew Its cert-manager Certificate. ServiceLB Was Stealing the HTTP-01 Challenge","Kubo Team",{"type":8,"value":9,"toc":266},"minimark",[10,15,24,31,69,80,84,90,99,102,111,126,130,136,153,169,192,196,202,208,233,236,243,247,250,253],[11,12,14],"h2",{"id":13},"_1-the-symptom-cert-managers-certificate-stuck-at-ready-false","1. The Symptom — cert-manager's Certificate Stuck at \"Ready: False\"",[16,17,18,19,23],"p",{},"When you hit a ",[20,21,22],"strong",{},"k3s cert-manager certificate not renewing"," situation, most teams first suspect DNS configuration or the cert-manager version. But the real culprit usually sits much closer to the metal.",[16,25,26],{},[27,28],"img",{"alt":29,"src":30},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure\u002Fsection01.webp",[16,32,33,34,38,39,42,43,50,51,54,55,58,59,58,62,58,65,68],{},"It's not uncommon to see ",[35,36,37],"code",{},"kubectl get certificate"," stuck at ",[35,40,41],{},"Ready: False"," indefinitely in a setup that auto-renews Let's Encrypt certificates via cert-manager. A ",[44,45,49],"a",{"href":46,"rel":47},"https:\u002F\u002Fgithub.com\u002Fcert-manager\u002Fcert-manager\u002Fissues\u002F6762",[48],"nofollow","cert-manager GitHub issue"," documents exactly this: on K3s with Traefik, the HTTP-01 challenge pod stays \"Progressing\" forever, and accessing it returns a 503 Service Unavailable. The logs show a ",[35,52,53],{},"propagation check failed"," error, and the resource dependency chain — ",[35,56,57],{},"Certificate"," → ",[35,60,61],{},"CertificateRequest",[35,63,64],{},"Order",[35,66,67],{},"Challenge"," — is stuck somewhere along the way.",[16,70,71,72,75,76,79],{},"Diagnosing this starts with walking through all four resources with ",[35,73,74],{},"kubectl describe",". But teams often re-check their Traefik configuration over and over and find nothing wrong. That's because, in most cases, the problem is neither Traefik nor cert-manager itself — it's ",[20,77,78],{},"ServiceLB, a component K3s ships by default",". This kind of K3s-specific pitfall — \"I configured it correctly, but something else was quietly occupying the resource\" — is much faster to spot when you already have visibility into your cluster's day-to-day state.",[11,81,83],{"id":82},"_2-root-cause-1-servicelb-occupies-ports-80443-and-blocks-the-challenge-pod","2. Root Cause 1 — ServiceLB Occupies Ports 80\u002F443 and Blocks the Challenge Pod",[16,85,86],{},[27,87],{"alt":88,"src":89},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure\u002Fsection02.webp",[16,91,92,93,98],{},"K3s ships a lightweight load balancer implementation called ServiceLB (formerly known as Klipper LB) that lets you use LoadBalancer-type Services out of the box. Per the ",[44,94,97],{"href":95,"rel":96},"https:\u002F\u002Fdocs.k3s.io\u002Fnetworking\u002Fnetworking-services",[48],"official K3s documentation",", ServiceLB creates a DaemonSet for every LoadBalancer-type Service and deploys a pod on each cluster node that directly occupies that service's port via hostPort.",[16,100,101],{},"In the default configuration, K3s's bundled Traefik claims ports 80 and 443 as a LoadBalancer service through this same mechanism. As a result, any other LoadBalancer service or hostPort pod requesting the same port number on the same node ends up conflicting and stuck in Pending.",[16,103,104,105,110],{},"As the ",[44,106,109],{"href":107,"rel":108},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fdiscussions\u002F9927",[48],"k3s-io\u002Fk3s discussion"," explains, ServiceLB pods themselves don't actually process traffic. They set up iptables rules on each node that map external IPs to the ClusterIP and forward traffic from node ports — all handled at the kernel level. In other words, \"what you see in the config\" and \"what's actually forwarding packets\" don't match, which is exactly why port-80 reachability issues are so hard to spot just by checking pod logs or Traefik's configuration.",[16,112,113,114,119,120,125],{},"cert-manager's HTTP-01 challenge, according to the ",[44,115,118],{"href":116,"rel":117},"https:\u002F\u002Fcert-manager.io\u002Fdocs\u002Fconfiguration\u002Facme\u002Fhttp01\u002F",[48],"cert-manager official documentation",", routes to a temporary acmesolver pod using your existing Ingress or Gateway configuration. If port-80 reachability is broken anywhere along that path, the challenge keeps failing silently. ",[44,121,124],{"href":122,"rel":123},"https:\u002F\u002Fletsencrypt.org\u002Fdocs\u002Fchallenge-types\u002F",[48],"Let's Encrypt's HTTP-01 challenge spec"," mandates that validation always happens on port 80 — the ACME standard doesn't allow specifying an arbitrary port. In other words, as long as port 80 reachability is broken, there's no alternate-port workaround.",[11,127,129],{"id":128},"_3-root-cause-2-traefik-ignores-the-challenge-ingress-due-to-an-ingressclassname-mismatch","3. Root Cause 2 — Traefik Ignores the Challenge Ingress Due to an ingressClassName Mismatch",[16,131,132],{},[27,133],{"alt":134,"src":135},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure\u002Fsection03.webp",[16,137,138,139,142,143,148,149,152],{},"Another commonly overlooked factor is the ",[35,140,141],{},"ingressClassName"," on the ACME challenge Ingress that cert-manager auto-generates. As the ",[44,144,147],{"href":145,"rel":146},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fservices-networking\u002Fingress\u002F",[48],"official Kubernetes documentation"," explains, an Ingress resource's ",[35,150,151],{},"spec.ingressClassName"," field determines which Ingress controller processes that resource, and in an environment without a default IngressClass defined, the behavior depends on the controller's implementation.",[16,154,155,156,161,162,165,166,168],{},"A ",[44,157,160],{"href":158,"rel":159},"https:\u002F\u002Fgithub.com\u002Fk3s-io\u002Fk3s\u002Fissues\u002F4765",[48],"k3s-io\u002Fk3s GitHub issue"," reports that when cert-manager's auto-generated challenge Ingress carries ",[35,163,164],{},"ingressClassName: traefik",", K3s's bundled Traefik sometimes fails to recognize it. The workaround is to manually remove ",[35,167,141],{}," from the Ingress resource, but that's only a stopgap — the same problem resurfaces the moment the Ingress is regenerated on the next renewal cycle.",[16,170,171,172,177,178,181,182,185,186,191],{},"The ",[44,173,176],{"href":174,"rel":175},"https:\u002F\u002Fcommunity.traefik.io\u002Ft\u002Fcert-manager-http-01-challenge-failing\u002F15575",[48],"Traefik Labs community forum"," also documents cases where Traefik fails to respond correctly to requests under ",[35,179,180],{},"\u002F.well-known\u002Facme-challenge\u002F"," and instead forwards them straight to the IngressRoute's backend service, as well as cases where an active HTTP-to-HTTPS redirect swallows the challenge request itself and causes it to fail. ",[20,183,184],{},"Root cause 1 (ServiceLB's port occupation) and root cause 2 (the ingressClassName mismatch) are independent problems, but in real-world incidents they often occur together"," — fixing only one of them may not be enough. Some managed Kubernetes offerings template this Ingress-controller\u002Fload-balancer alignment at provisioning time, and ",[44,187,190],{"href":188,"rel":189},"https:\u002F\u002Fkubo.hexabase.io\u002F",[48],"Kubo"," is one of them.",[11,193,195],{"id":194},"_4-the-fix-pinning-nodes-adjusting-ingressclassname-and-deciding-when-to-disable-servicelb","4. The Fix — Pinning Nodes, Adjusting ingressClassName, and Deciding When to Disable ServiceLB",[16,197,198],{},[27,199],{"alt":200,"src":201},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure\u002Fsection04.webp",[16,203,204,205,207],{},"As a quick fix, you can delete and recreate the stuck Certificate, CertificateRequest, Order, and Challenge resources while reviewing the ",[35,206,141],{}," setting — this restores service temporarily, but it's a symptomatic treatment that risks recurring on the next renewal cycle.",[16,209,210,211,214,215,218,219,222,223,226,227,232],{},"There are two directions for a permanent fix. One, per the ",[44,212,97],{"href":95,"rel":213},[48],", is to use the ",[35,216,217],{},"enablelb"," and ",[35,220,221],{},"lbpool"," node labels to pin ServiceLB to a specific set of nodes, reducing the chance of port conflicts. The other is to disable ServiceLB entirely with ",[35,224,225],{},"--disable=servicelb"," and switch to a full-fledged load balancer implementation designed for bare-metal\u002Fon-prem environments, such as ",[44,228,231],{"href":229,"rel":230},"https:\u002F\u002Fmetallb.io\u002F",[48],"MetalLB",". MetalLB integrates with your existing network equipment via BGP or L2 mode, delivering cloud-equivalent load balancing on-premises.",[16,234,235],{},"Which approach to choose depends on cluster size and operational maturity. For small clusters with few nodes and no complex networking requirements, node pinning is often sufficient. For production clusters that continuously run multiple LoadBalancer services, adopting MetalLB to eliminate the port-conflict risk at the root tends to be the more stable long-term choice.",[16,237,238,239,242],{},"This kind of \"it was working by default, but a specific component was quietly monopolizing a resource\" configuration mismatch is a pattern you'll run into repeatedly when operating K3s in production. ",[44,240,190],{"href":188,"rel":241},[48],"'s Captain UI gives visibility into this kind of cluster configuration state, making it easier to catch \"invisible occupiers\" like ServiceLB's port claims or IngressClass mismatches before they cause an incident.",[11,244,246],{"id":245},"_5-takeaway-k3ss-silently-skipped-defaults-can-become-the-spark-for-production-incidents","5. Takeaway — K3s's \"Silently Skipped Defaults\" Can Become the Spark for Production Incidents",[16,248,249],{},"K3s ships ServiceLB and Traefik by default as a lightweight Kubernetes distribution, which dramatically cuts down initial setup effort. But when those \"silently skipped defaults\" go unexamined and make it into production, they can lead to exactly the kind of hard-to-diagnose incident described here — a cert-manager certificate renewal failure.",[16,251,252],{},"If you run into a certificate that won't renew, don't stop at checking cert-manager itself or your DNS settings — also check the K3s-specific layer: ServiceLB's port occupation and IngressClass alignment.",[16,254,255,256,259,260,265],{},"If you'd rather reduce these kinds of operational pitfalls up front instead of wrestling with the high cost of EKS or AKS, K3s-based ",[44,257,190],{"href":188,"rel":258},[48]," is worth a look. It comes with Let's Encrypt certificate management built in, and Captain UI's visibility into configuration gaps can help you avoid incidents like the one described in this article. If you'd rather not step into the same trap, feel free to reach out via our ",[44,261,264],{"href":262,"rel":263},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[48],"contact page",".",{"title":267,"searchDepth":268,"depth":268,"links":269},"",2,[270,271,272,273,274],{"id":13,"depth":268,"text":14},{"id":82,"depth":268,"text":83},{"id":128,"depth":268,"text":129},{"id":194,"depth":268,"text":195},{"id":245,"depth":268,"text":246},"2026-09-25","cert-manager certificates on K3s stopped renewing, and neither Traefik nor cert-manager itself was the cause — ServiceLB was occupying ports 80\u002F443. Here's how the HTTP-01 challenge silently fails, and how to fix it for good.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure",{"title":5,"description":276},"blog\u002Fen\u002Fk3s-cert-manager-servicelb-http01-challenge-renewal-failure",[286,287,288,289,290,291],"k3s","kubernetes","cert-manager","servicelb","tls","lets-encrypt","g8NSrYpQaicFYqUupFLA-VSeQrB99Pa5BbQYKXzgWgE",[294,301,308,316,324,333],{"path":295,"title":296,"description":297,"date":298,"tags":299},"\u002Fblog\u002Fen\u002Fk3s-cert-manager-acme-renewal-pitfalls","A Saturday Morning K3s Outage from an Expired TLS Certificate: Designing ACME Automation That Actually Escapes Manual Renewal","Even with cert-manager installed, K3s production certificates can silently expire due to Issuer misconfiguration, failed DNS validation, Let's Encrypt rate limits, or Secrets that never get reloaded. This post covers the monitoring and design choices that turn ACME auto-renewal from 'installed' into 'actually working.'","2026-08-27",[286,287,288,290,291,300],"acme",{"path":302,"title":303,"description":304,"date":305,"tags":306},"\u002Fblog\u002Fen\u002Fkubernetes-certificate-management-cert-manager-process-debt","The Cert Renewal Took One Line of Code and Two Months of Meetings: Why Kubernetes Certificate Management Is a Process Problem, Not a Technical One","Kubernetes certificate management is technically a matter of days. What actually takes time is the organizational process of getting sign-off. Here's how cert-manager automates the technical side, and how to design away the operational debt that remains.","2026-08-09",[286,287,288,290,307],"security",{"path":309,"title":310,"description":311,"date":312,"tags":313},"\u002Fblog\u002Fen\u002Fcert-manager-automatic-tls","Automating TLS Certificate Management in Kubernetes with cert-manager","Automate TLS certificate issuance and renewal in Kubernetes using cert-manager. Covers Let's Encrypt, ACME challenges, Ingress integration, and best practices.","2026-05-27",[288,287,290,291,300,314,307,315],"cncf","certificates",{"path":317,"title":318,"description":319,"date":320,"tags":321},"\u002Fblog\u002Fen\u002Fk3s-pod-security-admission-restricted-servicelb-outage","We Enabled Restricted and ServiceLB Died: The K3s Pod Security Admission Gap No Generic Guide Mentions","Enabling the restricted Pod Security Admission profile on K3s following standard Kubernetes guides can silently kill ServiceLB. Here's why, backed by K3s's own CIS hardening guide, and how to roll it out safely.","2026-10-16",[286,287,322,323,289,307],"pod-security-admission","pod-security-standards",{"path":325,"title":326,"description":327,"date":328,"tags":329},"\u002Fblog\u002Fen\u002Fk3s-networkpolicy-dual-stack-kube-router-ipv6-gap","\"K3s Ignores NetworkPolicy\" Was a Myth. Traffic Still Leaked Through — the ServiceLB Blind Spot","K3s enforces NetworkPolicy by default via its built-in kube-router controller. Yet traffic via LoadBalancer still leaked through — the cause was ServiceLB's implementation combined with a permanent limitation of the NetworkPolicy API. Here's how to detect and fix it.","2026-09-24",[286,287,330,289,331,332],"networkpolicy","kube-router","network-security",{"path":334,"title":335,"description":336,"date":337,"tags":338},"\u002Fblog\u002Fen\u002Fk3s-traefik-servicelb-ingress-nginx-port-conflict","You Disabled Traefik on K3s, But ingress-nginx Still Can't Grab Port 443. Meet ServiceLB, the Roommate You Forgot About","You ran the K3s disable-traefik flag but ingress-nginx still can't bind port 443. The culprit is ServiceLB (klipper-lb), another default K3s component. Here's how to diagnose it, disable it correctly, and what changes if you're running MetalLB.","2026-09-19",[286,287,339,289,340,341],"traefik","ingress-nginx","load-balancer",1792592632516]