HPA Replica Count Flaps Every Few Minutes. The Root Cause Wasn't Metrics — It Was a Missing scaleDown Stabilization Window

A production K3s cluster kept swinging its HPA replica count between 2 and 6 every 5 minutes

On a production K3s cluster, a workload with HPA (HorizontalPodAutoscaler) enabled started swinging its replica count between 2 and 6 pods every five minutes. Looking at the dashboards, request volume and CPU load were both flat — nothing looked obviously wrong. Yet the replica count kept oscillating like a wave.

This is a textbook case of "HPA scaling flap." Digging deeper, the metrics pipeline itself was fine — the root cause was that the HPA's behavior field had never been explicitly configured. The default scaleUp/scaleDown behavior turned out to be more aggressive than expected. If you'd rather not have to build your own dashboard to notice this kind of quirk, Kubo's Captain UI visualizes scaling trends out of the box, which shortens the time it takes to spot problems like this.

This article breaks down the three actual causes we found (asymmetric behavior defaults, a CPU requests denominator problem, and latency from metrics sync intervals) and walks through the concrete fix for a K3s environment.

Cause 1: The scaleUp/scaleDown defaults are asymmetric, and behavior was never configured

According to the Kubernetes official documentation, HPA's scaleUp and scaleDown each have different default stabilization windows (stabilizationWindowSeconds).

• scaleUp: stabilization window of 0 seconds (instant), selectPolicy is Max (picks the most aggressive recommendation)
• scaleDown: stabilization window of 300 seconds (5 minutes), selectPolicy is Max

In other words, the replica count increases the instant load rises, but it doesn't decrease right away when load drops. That much is intentional, "safety-first" design. But if you never configure the behavior field and load fluctuates even slightly, scaleUp's "instant" reaction keeps overriding scaleDown's "5-minute wait" effect, and the replica count gradually starts to oscillate.

What the stabilization window actually does

The stabilization window isn't a simple "delay timer." Per the official documentation, it picks the "most conservative value" from the recommended replica counts recorded within the window. If the window is too short, there isn't enough history to choose a conservative value from, and the oscillation goes unchecked.

Cause 2: "Optimizing" CPU requests actually made scaling more sensitive

Many teams tighten resources.requests.cpu to match real usage as a cost-cutting measure. But this can make HPA flapping worse.

This is exactly the "percentage math problem" pointed out by the ScaleOps blog. HPA's CPU utilization is calculated as:

utilization = current CPU usage ÷ CPU requests × 100

Even if actual workload consumption hasn't changed, the moment you optimize requests from 200m to 100m, the denominator is cut in half, so apparent utilization jumps from 75% to 150%. HPA misreads this as a sudden spike in load and triggers an unnecessary scale-out. It's a classic case where a cost-saving requests optimization ironically destabilizes scaling.

Cause 3: The sync interval between metrics-server and HPA was stacking up latency

K3s bundles metrics-server by default, powering both kubectl top and HPA's CPU/memory metrics. That's convenient, but it's worth understanding the latency involved.

Per the official metrics-server repository's FAQ, the default metric-resolution is 60 seconds, but setting it below 15 seconds isn't recommended, since 15 seconds is the resolution at which kubelet itself calculates metrics. Many production environments run with metric-resolution set to 15 seconds.

Separately, the HPA controller has its own sync interval: per the kube-controller-manager reference, the default for --horizontal-pod-autoscaler-sync-period is 15 seconds. The official metrics-server site also explains that this scrape interval and HPA's polling interval run independently of each other.

That means, at minimum, 15-30 seconds of latency stacks up between a load spike happening and HPA actually "noticing" it. If load keeps fluctuating during that window, HPA is always deciding based on a "slightly stale" view of the world, which further fuels the flapping.

The fix: set explicit policies and selectPolicy in behavior to stop the flapping

Based on the practical recommendations from the MeteorOps blog, here's how we rewrote the HPA manifest on K3s.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 25
        periodSeconds: 60
      selectPolicy: Min

There are three key points here.

• Set the scaleUp stabilization window to 60 seconds: stop reacting instantly, so short-lived spikes don't trigger overreactions
• Cap scaleDown at 25% via a Percent policy: remove replicas gradually instead of cutting a large batch at once
• Switch selectPolicy to Min: when multiple policies conflict, pick the most conservative (least disruptive) value

Alongside this, it's worth giving CPU requests a safety margin over measured usage and targeting an averageUtilization of around 60-75%, per MeteorOps' recommended values. Keeping production minReplicas at 3 or more also helps absorb scale-up latency with baseline capacity.

Don't relax just because you set it — stabilizationWindowSeconds has a known quirk

Configuring behavior isn't the end of the story. GitHub Issue #101371 on kubernetes/kubernetes reports cases where, even with stabilizationWindowSeconds: 300 configured, scale-down actually happens after just 1-2 minutes. It's a known behavioral quirk logged as a bug under SIG Autoscaling.

In other words, "I set it to 300 seconds, so it's safe" isn't a safe assumption on its own — you need to keep monitoring the actual behavior on a dashboard. Configuration and observed behavior are two different things.

With Kubo's Captain UI, you can watch replica count trends on a real-time graph, making it easier to visually confirm whether things are behaving as configured. Catching this kind of drift early, without having to build your own dashboard, is a real operational advantage.

Summary

HPA replica count oscillation is more often caused by insufficient behavior design than by broken metrics. In this case, three factors were compounding each other:

1. The asymmetric defaults of scaleUp (0 seconds, instant) and scaleDown (300 seconds) were never overridden with an explicit behavior configuration
2. Optimizing CPU requests changed the denominator of utilization, inflating the apparent usage percentage
3. The sync intervals of metrics-server and the HPA controller (roughly 15 seconds each) stacked up as latency

The fix comes down to explicitly setting stabilizationWindowSeconds, policies (Percent/Pods), and selectPolicy, and tuning CPU requests to match real usage. But it doesn't end with configuration — you need to keep watching how it actually behaves.

With K3s-based Kubo, you can apply HPA behavior templates via AI-Driven Deployment while visualizing scaling trends in Captain UI. A comparable EKS or AKS setup often runs around ¥82,700-85,710 (roughly $550-570) per month, while Kubo starts at ¥48,000 (roughly $320) for a full-spec Kubernetes setup, as shown on the pricing page. If you'd rather not run production without noticing HPA misconfigurations like this, get in touch for a consultation.