Question Your discoveryInterval: Why K3s HPA Keeps Scaling on Stale Metrics from Prometheus Adapter

1. When "Metrics Look Fine" But HPA Won't React

If you run HPA (Horizontal Pod Autoscaler) alongside Prometheus Adapter on K3s, you'll eventually hit a strange incident: CPU usage and custom metrics look perfectly normal, yet Pods simply won't scale out.

Run kubectl describe hpa and you may see a target value displayed as <unknown>/50. According to the official Kubernetes documentation, this happens when a Pod's resource requests aren't set, metrics-server isn't installed, or a query to one of the metrics APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io) fails.

What's even trickier is that stale metrics aren't the only failure mode — a broken metric can actually trigger runaway over-scaling. A real incident reported on the official prometheus-adapter repository describes an HPA configured with CPU, memory, and a custom metric: when the custom metric started returning 404, the HPA lost control and the Deployment scaled up far beyond what was needed. "Stale metrics" and "broken metrics" are two distinct failure modes, and both trace back to the state of the Prometheus Adapter pipeline.

The operational burden of maintaining this metrics pipeline yourself never really goes away if you're running Prometheus Adapter on your own. With Kubo, Prometheus-based monitoring comes standard on a managed K3s foundation, so at least the burden of maintaining the monitoring stack itself is lifted.

2. The Real Culprit: A Three-Layer Delay Between Metric and HPA

Most incidents like this aren't actually bugs in HPA itself — they come from accumulated delay as a metric passes through three layers: Prometheus → Prometheus Adapter → HPA.

1. Prometheus's scrape interval: Per the official Prometheus documentation, the global default for scrape_interval is one minute. Unless overridden per target, metrics only refresh at this cadence.
2. Prometheus Adapter's freshness parameters: According to the official prometheus-adapter README, --metrics-relist-interval (default 10 minutes) controls how often the adapter refreshes its cache of which metrics exist from Prometheus, while --metrics-max-age (defaults to the same value as the relist interval) is the time window the adapter uses to decide whether a metric is still "present." The official README informally calls this window the "discovery interval."
3. HPA's polling interval: As the official Kubernetes documentation states, kube-controller-manager's --horizontal-pod-autoscaler-sync-period defaults to 15 seconds. K3s runs on this same default unless you explicitly change it.

The easy-to-miss asymmetry: HPA dutifully checks every 15 seconds, but the value it's looking at can be as stale as the metrics-relist-interval/metrics-max-age window allows. With default settings, the adapter's freshness window (on the order of 10 minutes) is far coarser than HPA's polling interval (15 seconds) — so for a sudden traffic spike, it's entirely plausible for Pods to take minutes to scale out even though "the metric looked fine."

3. What Happens When discoveryInterval (metrics-max-age) Is Shorter Than the Collection Interval

The key thing to watch with discoveryInterval (really, --metrics-max-age) is its relationship to Prometheus's scrape interval. The official prometheus-adapter README warns explicitly: "metrics-max-age should be equal to or larger than your Prometheus scrape interval, or your metrics will occasionally disappear from the adapter."

In other words, if you set discoveryInterval shorter than necessary, a slightly delayed Prometheus scrape can push a metric outside the adapter's window, making HPA think the metric doesn't exist at all. The metricsQuery rate window defined in prometheus-adapter's config.md (e.g. rate(http_requests_total[2m])) carries a similar trade-off: too short and it's overly sensitive to momentary gaps, too long and it smooths things out so much that HPA reacts late.

If you've deployed the Prometheus Operator or kube-prometheus-stack on K3s with default settings, you rarely get a reason to think about this relationship between discoveryInterval and the scrape interval — which is exactly why it should be the first thing you check when HPA starts behaving erratically.

4. Why Adding KEDA Doesn't Make Things Faster

"If HPA is slow, just add KEDA" is only half right. Per the official KEDA ScaledObject specification, pollingInterval (default 30 seconds) is the interval at which KEDA checks each trigger; while the replica count is 0, KEDA polls at this interval to decide whether to scale.

However, once the replica count is between 1 and N, the regular HPA also evaluates metrics periodically, at a frequency governed by kube-controller-manager's --horizontal-pod-autoscaler-sync-period (default 15 seconds). In other words, during the 1-to-N scaling phase, it's ultimately HPA that's reading metrics straight from Prometheus Adapter — so the freshness problems caused by discoveryInterval/metrics-relist-interval are not resolved just by adding KEDA. What KEDA adds is strictly the 0-to-1 activation decision, a separate layer entirely.

Incidentally, the Kubernetes community is discussing a mechanism (KEP-5679) that would use a static fallback replica count if external metric retrieval keeps failing for a set duration. This only addresses "retrieval failure," not "retrieval succeeds but the value is stale," which is the subject of this article. For now, aligning the interval settings across the pipeline is still something you have to do yourself.

It's easy to forget that both KEDA and HPA end up putting the burden of interval design on you. Kubo Cloud ships Prometheus + Grafana standard alongside GitOps and Helm chart support, so you can manage KEDA/HPA configuration changes through ArgoCD/Flux with full version control.

5. A Diagnostic Checklist for K3s/Kubo Environments

Here's a priority-ordered checklist for diagnosing an "HPA won't react" incident.

1. Check the Conditions: Run kubectl describe hpa <name> and check whether the target value shows <unknown> or whether there's an error in the Conditions section. As the official Datadog troubleshooting documentation notes, FailedGetExternalMetric typically points to missing RBAC or a connectivity problem with the metrics provider, while FailedComputeMetricsReplicas tends to show up when the metrics provider itself isn't running.
2. Check APIService status: Run kubectl get apiservices | grep metrics and confirm v1beta1.custom.metrics.k8s.io / v1beta1.external.metrics.k8s.io show Available=True.
3. Confirm the metric is registered directly: Run kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 to see whether the metric you're targeting actually shows up in the list.
4. Cross-check the interval settings: Compare prometheus-adapter's launch flags --metrics-relist-interval / --metrics-max-age against your actual Prometheus scrape_interval. If the latter is larger than the former, metrics can periodically disappear.
5. Confirm resource requests are set: For CPU-based HPA, as the official Kubernetes documentation notes, utilization simply can't be calculated if a Pod doesn't have resource requests defined.

If you're building and running Prometheus + Grafana yourself on K3s, tuning and monitoring these intervals becomes an ongoing operational burden for your SRE team. With a managed K3s environment like Kubo that ships Prometheus + Grafana standard, you no longer need to spend time building and keeping the monitoring stack itself up to date, freeing you up to focus on this kind of "freshness design" tuning instead.

6. Summary

Most "HPA looks fine but won't react" incidents aren't caused by a bug in HPA itself — they come from three independent time settings failing to line up: Prometheus's scrape interval, Prometheus Adapter's metrics-relist-interval/metrics-max-age, and HPA's sync-period. discoveryInterval (metrics-max-age), unassuming as it looks, should be the first setting you suspect during an investigation.

Adding KEDA speeds up the 0-to-1 scaling decision, but in the 1-to-N phase, HPA still depends on metrics from Prometheus Adapter — so it doesn't solve the freshness problem discussed here. Learning to question "how fast the metric arrives," not just "what the metric's value is," is the key to preventing your next autoscaling incident on K3s/Kubernetes.

If you'd rather not keep accumulating this kind of operational know-how on your own, Kubo Cloud ships a Prometheus + Grafana monitoring stack standard on a K3s foundation, letting you run production-grade Kubernetes at less than half the cost of EKS/AKS. Instead of wiring up every interval setting by hand, you can also just get in touch to talk through your setup.