[{"data":1,"prerenderedAt":442},["ShallowReactive",2],{"blog-en-k3s-hpa-prometheus-adapter-discovery-interval-stale-metrics":3,"blog-related-en-k3s-hpa-prometheus-adapter-discovery-interval-stale-metrics":397,"blog-en-k3s-hpa-prometheus-adapter-discovery-interval-stale-metrics-alt":385},{"id":4,"title":5,"author":6,"body":7,"date":379,"description":380,"extension":381,"image":382,"locale":383,"meta":384,"navigation":385,"path":386,"seo":387,"stem":388,"tags":389,"__hash__":396},"blog\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics.md","Question Your discoveryInterval: Why K3s HPA Keeps Scaling on Stale Metrics from Prometheus Adapter","Kubo Team",{"type":8,"value":9,"toc":369},"minimark",[10,15,23,26,57,71,80,84,87,144,151,155,161,175,192,195,199,205,217,229,238,246,250,256,259,339,346,350,353,356],[11,12,14],"h2",{"id":13},"_1-when-metrics-look-fine-but-hpa-wont-react","1. When \"Metrics Look Fine\" But HPA Won't React",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics\u002Fsection01.webp",[16,24,25],{},"If you run HPA (Horizontal Pod Autoscaler) alongside Prometheus Adapter on K3s, you'll eventually hit a strange incident: CPU usage and custom metrics look perfectly normal, yet Pods simply won't scale out.",[16,27,28,29,33,34,37,38,45,46,49,50,49,53,56],{},"Run ",[30,31,32],"code",{},"kubectl describe hpa"," and you may see a target value displayed as ",[30,35,36],{},"\u003Cunknown>\u002F50",". According to the ",[39,40,44],"a",{"href":41,"rel":42},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Frun-application\u002Fhorizontal-pod-autoscale\u002F",[43],"nofollow","official Kubernetes documentation",", this happens when a Pod's resource requests aren't set, metrics-server isn't installed, or a query to one of the metrics APIs (",[30,47,48],{},"metrics.k8s.io",", ",[30,51,52],{},"custom.metrics.k8s.io",[30,54,55],{},"external.metrics.k8s.io",") fails.",[16,58,59,60,64,65,70],{},"What's even trickier is that stale metrics aren't the only failure mode — ",[61,62,63],"strong",{},"a broken metric can actually trigger runaway over-scaling",". A ",[39,66,69],{"href":67,"rel":68},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fprometheus-adapter\u002Fissues\u002F636",[43],"real incident reported"," on the official prometheus-adapter repository describes an HPA configured with CPU, memory, and a custom metric: when the custom metric started returning 404, the HPA lost control and the Deployment scaled up far beyond what was needed. \"Stale metrics\" and \"broken metrics\" are two distinct failure modes, and both trace back to the state of the Prometheus Adapter pipeline.",[16,72,73,74,79],{},"The operational burden of maintaining this metrics pipeline yourself never really goes away if you're running Prometheus Adapter on your own. With ",[39,75,78],{"href":76,"rel":77},"https:\u002F\u002Fkubo.hexabase.io\u002F",[43],"Kubo",", Prometheus-based monitoring comes standard on a managed K3s foundation, so at least the burden of maintaining the monitoring stack itself is lifted.",[11,81,83],{"id":82},"_2-the-real-culprit-a-three-layer-delay-between-metric-and-hpa","2. The Real Culprit: A Three-Layer Delay Between Metric and HPA",[16,85,86],{},"Most incidents like this aren't actually bugs in HPA itself — they come from accumulated delay as a metric passes through three layers: Prometheus → Prometheus Adapter → HPA.",[88,89,90,107,126],"ol",{},[91,92,93,96,97,102,103,106],"li",{},[61,94,95],{},"Prometheus's scrape interval",": Per the ",[39,98,101],{"href":99,"rel":100},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fprometheus\u002Flatest\u002Fconfiguration\u002Fconfiguration\u002F",[43],"official Prometheus documentation",", the global default for ",[30,104,105],{},"scrape_interval"," is one minute. Unless overridden per target, metrics only refresh at this cadence.",[91,108,109,112,113,49,118,121,122,125],{},[61,110,111],{},"Prometheus Adapter's freshness parameters",": According to the ",[39,114,117],{"href":115,"rel":116},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fprometheus-adapter\u002Fblob\u002Fmaster\u002FREADME.md",[43],"official prometheus-adapter README",[30,119,120],{},"--metrics-relist-interval"," (default 10 minutes) controls how often the adapter refreshes its cache of which metrics exist from Prometheus, while ",[30,123,124],{},"--metrics-max-age"," (defaults to the same value as the relist interval) is the time window the adapter uses to decide whether a metric is still \"present.\" The official README informally calls this window the \"discovery interval.\"",[91,127,128,131,132,135,136,139,140,143],{},[61,129,130],{},"HPA's polling interval",": As the ",[39,133,44],{"href":41,"rel":134},[43]," states, ",[30,137,138],{},"kube-controller-manager","'s ",[30,141,142],{},"--horizontal-pod-autoscaler-sync-period"," defaults to 15 seconds. K3s runs on this same default unless you explicitly change it.",[16,145,146,147,150],{},"The easy-to-miss asymmetry: ",[61,148,149],{},"HPA dutifully checks every 15 seconds, but the value it's looking at can be as stale as the metrics-relist-interval\u002Fmetrics-max-age window allows."," With default settings, the adapter's freshness window (on the order of 10 minutes) is far coarser than HPA's polling interval (15 seconds) — so for a sudden traffic spike, it's entirely plausible for Pods to take minutes to scale out even though \"the metric looked fine.\"",[11,152,154],{"id":153},"_3-what-happens-when-discoveryinterval-metrics-max-age-is-shorter-than-the-collection-interval","3. What Happens When discoveryInterval (metrics-max-age) Is Shorter Than the Collection Interval",[16,156,157],{},[19,158],{"alt":159,"src":160},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics\u002Fsection02.webp",[16,162,163,164,166,167,170,171,174],{},"The key thing to watch with discoveryInterval (really, ",[30,165,124],{},") is its relationship to Prometheus's scrape interval. The ",[39,168,117],{"href":115,"rel":169},[43]," warns explicitly: \"",[30,172,173],{},"metrics-max-age"," should be equal to or larger than your Prometheus scrape interval, or your metrics will occasionally disappear from the adapter.\"",[16,176,177,178,181,182,187,188,191],{},"In other words, if you set discoveryInterval shorter than necessary, a slightly delayed Prometheus scrape can push a metric outside the adapter's window, making HPA think the metric doesn't exist at all. The ",[30,179,180],{},"metricsQuery"," rate window defined in ",[39,183,186],{"href":184,"rel":185},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fprometheus-adapter\u002Fblob\u002Fmaster\u002Fdocs\u002Fconfig.md",[43],"prometheus-adapter's config.md"," (e.g. ",[30,189,190],{},"rate(http_requests_total[2m])",") carries a similar trade-off: too short and it's overly sensitive to momentary gaps, too long and it smooths things out so much that HPA reacts late.",[16,193,194],{},"If you've deployed the Prometheus Operator or kube-prometheus-stack on K3s with default settings, you rarely get a reason to think about this relationship between discoveryInterval and the scrape interval — which is exactly why it should be the first thing you check when HPA starts behaving erratically.",[11,196,198],{"id":197},"_4-why-adding-keda-doesnt-make-things-faster","4. Why Adding KEDA Doesn't Make Things Faster",[16,200,201],{},[19,202],{"alt":203,"src":204},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics\u002Fsection03.webp",[16,206,207,208,49,213,216],{},"\"If HPA is slow, just add KEDA\" is only half right. Per the ",[39,209,212],{"href":210,"rel":211},"https:\u002F\u002Fkeda.sh\u002Fdocs\u002F2.20\u002Freference\u002Fscaledobject-spec",[43],"official KEDA ScaledObject specification",[30,214,215],{},"pollingInterval"," (default 30 seconds) is the interval at which KEDA checks each trigger; while the replica count is 0, KEDA polls at this interval to decide whether to scale.",[16,218,219,220,139,222,224,225,228],{},"However, once the replica count is between 1 and N, the regular HPA also evaluates metrics periodically, at a frequency governed by ",[30,221,138],{},[30,223,142],{}," (default 15 seconds). In other words, ",[61,226,227],{},"during the 1-to-N scaling phase, it's ultimately HPA that's reading metrics straight from Prometheus Adapter"," — so the freshness problems caused by discoveryInterval\u002Fmetrics-relist-interval are not resolved just by adding KEDA. What KEDA adds is strictly the 0-to-1 activation decision, a separate layer entirely.",[16,230,231,232,237],{},"Incidentally, the Kubernetes community is discussing a mechanism (",[39,233,236],{"href":234,"rel":235},"https:\u002F\u002Fwww.kubernetes.dev\u002Fresources\u002Fkeps\u002F5679\u002F",[43],"KEP-5679",") that would use a static fallback replica count if external metric retrieval keeps failing for a set duration. This only addresses \"retrieval failure,\" not \"retrieval succeeds but the value is stale,\" which is the subject of this article. For now, aligning the interval settings across the pipeline is still something you have to do yourself.",[16,239,240,241,245],{},"It's easy to forget that both KEDA and HPA end up putting the burden of interval design on you. ",[39,242,244],{"href":76,"rel":243},[43],"Kubo Cloud"," ships Prometheus + Grafana standard alongside GitOps and Helm chart support, so you can manage KEDA\u002FHPA configuration changes through ArgoCD\u002FFlux with full version control.",[11,247,249],{"id":248},"_5-a-diagnostic-checklist-for-k3skubo-environments","5. A Diagnostic Checklist for K3s\u002FKubo Environments",[16,251,252],{},[19,253],{"alt":254,"src":255},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics\u002Fsection04.webp",[16,257,258],{},"Here's a priority-ordered checklist for diagnosing an \"HPA won't react\" incident.",[88,260,261,289,306,315,329],{},[91,262,263,266,267,270,271,274,275,280,281,284,285,288],{},[61,264,265],{},"Check the Conditions",": Run ",[30,268,269],{},"kubectl describe hpa \u003Cname>"," and check whether the target value shows ",[30,272,273],{},"\u003Cunknown>"," or whether there's an error in the Conditions section. As the ",[39,276,279],{"href":277,"rel":278},"https:\u002F\u002Fdocs.datadoghq.com\u002Fcontainers\u002Ftroubleshooting\u002Fhpa\u002F",[43],"official Datadog troubleshooting documentation"," notes, ",[30,282,283],{},"FailedGetExternalMetric"," typically points to missing RBAC or a connectivity problem with the metrics provider, while ",[30,286,287],{},"FailedComputeMetricsReplicas"," tends to show up when the metrics provider itself isn't running.",[91,290,291,266,294,297,298,301,302,305],{},[61,292,293],{},"Check APIService status",[30,295,296],{},"kubectl get apiservices | grep metrics"," and confirm ",[30,299,300],{},"v1beta1.custom.metrics.k8s.io"," \u002F ",[30,303,304],{},"v1beta1.external.metrics.k8s.io"," show Available=True.",[91,307,308,266,311,314],{},[61,309,310],{},"Confirm the metric is registered directly",[30,312,313],{},"kubectl get --raw \u002Fapis\u002Fcustom.metrics.k8s.io\u002Fv1beta1"," to see whether the metric you're targeting actually shows up in the list.",[91,316,317,320,321,301,323,325,326,328],{},[61,318,319],{},"Cross-check the interval settings",": Compare prometheus-adapter's launch flags ",[30,322,120],{},[30,324,124],{}," against your actual Prometheus ",[30,327,105],{},". If the latter is larger than the former, metrics can periodically disappear.",[91,330,331,334,335,338],{},[61,332,333],{},"Confirm resource requests are set",": For CPU-based HPA, as the ",[39,336,44],{"href":41,"rel":337},[43]," notes, utilization simply can't be calculated if a Pod doesn't have resource requests defined.",[16,340,341,342,345],{},"If you're building and running Prometheus + Grafana yourself on K3s, tuning and monitoring these intervals becomes an ongoing operational burden for your SRE team. With a managed K3s environment like ",[39,343,78],{"href":76,"rel":344},[43]," that ships Prometheus + Grafana standard, you no longer need to spend time building and keeping the monitoring stack itself up to date, freeing you up to focus on this kind of \"freshness design\" tuning instead.",[11,347,349],{"id":348},"_6-summary","6. Summary",[16,351,352],{},"Most \"HPA looks fine but won't react\" incidents aren't caused by a bug in HPA itself — they come from three independent time settings failing to line up: Prometheus's scrape interval, Prometheus Adapter's metrics-relist-interval\u002Fmetrics-max-age, and HPA's sync-period. discoveryInterval (metrics-max-age), unassuming as it looks, should be the first setting you suspect during an investigation.",[16,354,355],{},"Adding KEDA speeds up the 0-to-1 scaling decision, but in the 1-to-N phase, HPA still depends on metrics from Prometheus Adapter — so it doesn't solve the freshness problem discussed here. Learning to question \"how fast the metric arrives,\" not just \"what the metric's value is,\" is the key to preventing your next autoscaling incident on K3s\u002FKubernetes.",[16,357,358,359,362,363,368],{},"If you'd rather not keep accumulating this kind of operational know-how on your own, ",[39,360,244],{"href":76,"rel":361},[43]," ships a Prometheus + Grafana monitoring stack standard on a K3s foundation, letting you run production-grade Kubernetes at less than half the cost of EKS\u002FAKS. Instead of wiring up every interval setting by hand, you can also just ",[39,364,367],{"href":365,"rel":366},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[43],"get in touch"," to talk through your setup.",{"title":370,"searchDepth":371,"depth":371,"links":372},"",2,[373,374,375,376,377,378],{"id":13,"depth":371,"text":14},{"id":82,"depth":371,"text":83},{"id":153,"depth":371,"text":154},{"id":197,"depth":371,"text":198},{"id":248,"depth":371,"text":249},{"id":348,"depth":371,"text":349},"2026-10-12","K3s HPA won't scale even though CPU usage looks normal. The culprit is a mismatch between Prometheus Adapter's metrics-relist-interval\u002Fmetrics-max-age and the HPA sync-period. Here's the three-layer delay chain and how to diagnose it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics",{"title":5,"description":380},"blog\u002Fen\u002Fk3s-hpa-prometheus-adapter-discovery-interval-stale-metrics",[390,391,392,393,394,395],"k3s","kubernetes","hpa","prometheus-adapter","autoscaling","observability","21GzHrEXRvZ8WDdeGoJQTyYA32ANImcQHNcWblXPm_E",[398,406,413,422,428,435],{"path":399,"title":400,"description":401,"date":402,"tags":403},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[390,391,392,404,394,405],"containerresource","service-mesh",{"path":407,"title":408,"description":409,"date":410,"tags":411},"\u002Fblog\u002Fen\u002Fkubernetes-hpa-vpa-autoscaling-conflict-design","\"Running Both Feels Safer\" Is the Trap. Why Combining HPA and VPA in Kubernetes Can Send Autoscaling Into a Spiral","Enabling both HPA and VPA in Kubernetes can make two well-intentioned automations fight each other, sending autoscaling into an endless spiral. Here's why the conflict happens, and how to combine them safely with updateMode and metric separation.","2026-10-07",[390,391,392,412,394],"vpa",{"path":414,"title":415,"description":416,"date":417,"tags":418},"\u002Fblog\u002Fen\u002Fk3s-keda-hpa-event-driven-autoscaling-decision","CPU Usage Looks Normal, But Requests Are Piling Up. When to Add KEDA to HPA in Kubernetes","KEDA and HPA aren't an either-or choice. When event-driven workloads grow on K3s\u002FKubernetes, here's why HPA alone hits its limits and how to decide when to add KEDA alongside it.","2026-10-03",[390,391,419,392,394,420,421],"keda","event-driven-scaling","scaledobject",{"path":423,"title":424,"description":425,"date":426,"tags":427},"\u002Fblog\u002Fen\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","KEDA vs HPA on K3s: A Decision Framework for Event-Driven Autoscaling","When CPU\u002Fmemory-based HPA on K3s can't keep up with event-driven workloads, should you extend HPA or adopt KEDA, the CNCF-graduated project? Here's a 3-point decision framework, plus what to watch for on K3s.","2026-09-10",[390,391,392,419,394],{"path":429,"title":430,"description":431,"date":432,"tags":433},"\u002Fblog\u002Fen\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral","Your HPA Config Is Correct. So Why Won't Pods Scale? The Real Cause of 'Unknown Metrics' in K3s and the HPA\u002FVPA Death Spiral","You set up an HPA in K3s and Pods still won't scale. The usual culprits are unset resource requests silently disabling metrics, and a self-reinforcing HPA\u002FVPA feedback loop. Here's how to diagnose and fix it.","2026-09-02",[390,391,392,412,394,434],"metrics-server",{"path":436,"title":437,"description":438,"date":439,"tags":440},"\u002Fblog\u002Fen\u002Fk3s-vpa-inplaceorrecreate-pod-resize-cgroup","VPA InPlaceOrRecreate Didn't Stop the Restart. Testing InPlacePodVerticalScaling on K3s","Does VPA's InPlaceOrRecreate mode and In-Place Pod Resize really avoid Pod restarts? We test QoS class constraints and the cgroup v1 trap on K3s with real configs and kubectl commands.","2026-10-17",[390,391,412,394,441],"cgroup",1792592635842]