The 5 Minutes Between a Scale-Out Signal and a Running Pod: Why Cluster Autoscaler Can't Add Nodes Fast Enough

1. Why Pods Sit Pending for Minutes Even Though HPA Is Working Correctly

Check the HPA metrics and the replica count has indeed gone up. But run kubectl get pods and the new Pods are all sitting in Pending, unmoving for minutes. No errors show up in the deployment logs either. In most cases, HPA isn't broken — a separate layer, node addition by Cluster Autoscaler, simply hasn't caught up.

HPA and Cluster Autoscaler both get lumped together under "autoscaling," but they operate at completely different layers. HPA adjusts the number of Pods; Cluster Autoscaler adjusts the number of nodes. According to the official Kubernetes documentation, the default stabilization window (stabilizationWindowSeconds) for HPA scale-up is 0 seconds — meaning it's designed to react to metric changes almost instantly.

Cluster Autoscaler, on the other hand, first has to detect that existing nodes have no room for new Pods (they become unschedulable), then request a new node from the cloud provider, and wait for that node to boot and join the cluster. This gap between an instantly-reacting HPA and a node addition process that takes minutes is what makes Pods appear to hang in mid-air.

2. Breaking Down the 5-Minute Delay Into Numbers

"A few minutes" isn't specific enough to act on. Break it down, and the delay turns out to be the sum of three sequential stages.

HPA's reaction time (roughly 1-3 minutes)

Because the metrics-server's collection interval overlaps with the HPA controller's sync period, it typically takes 1-3 minutes from an actual load increase until the replica count is updated. Per the behavior field description in the official Kubernetes documentation, the default scale-up policy is fairly aggressive: every 15-second period, it adds either "100% of current replicas" or "4 Pods," whichever is larger.

Cluster Autoscaler's reaction time (under 30 seconds)

Per the cluster-autoscaler FAQ, Cluster Autoscaler scans for unschedulable Pods every --scan-interval (10 seconds by default). Komodor's explainer also notes that the scale-up decision itself is reactively triggered "once the number of pending/unschedulable pods increases," and this detection step usually completes within 30 seconds.

Node provisioning on the cloud side (3-4 minutes, sometimes more)

This is where most of the time goes. Booting the VM instance, the OS, kubelet, and registering with the cluster — plus cloud-specific factors like the ones described in Google Cloud's official troubleshooting guide — typically add up to 3-4 minutes. Add it all up, and roughly 5 minutes total is a reasonable rule of thumb from HPA detection to a Pod actually running.

The cluster-autoscaler FAQ also documents --max-node-provision-time (15 minutes by default): if a node isn't registered within that window, it's excluded from the scale-up simulation and a different node group is tried instead. Conversely, --scale-down-delay-after-add (roughly 10 minutes by default) is a safety margin that prevents a newly added node from being removed again right away.

3. A Checklist for When "There Should Be Room, But It's Not Scaling Up"

If nothing changes after five minutes, suspect a configuration problem rather than simple latency. Google Cloud's official troubleshooting guide organizes the typical reasons a GKE scale-up fails into specific error codes.

The same guide notes that Cluster Autoscaler stops scaling up as a safety measure once more than 45% of nodes are unhealthy. Running kubectl describe configmap cluster-autoscaler-status -n kube-system to check the current decision reason is a useful diagnostic step regardless of cloud provider.

In self-hosted or edge environments like K3s, the "cloud API quota" factor doesn't apply — instead, the spare capacity of existing nodes becomes the focus. Per K3s's official resource profiling documentation, when a new agent node joins the cluster, the server's (control plane's) CPU usage has been observed to spike roughly 20% above baseline, and the limiting factor for scaling is described as "CPU, not RAM." If node addition feels slow on-prem or at the edge, it's worth checking the control plane's own CPU headroom, not just conditions on the destination cloud. Whether this kind of diagnosis can be visualized on a dashboard directly affects operational load — Kubo's Captain UI ships with screens for visualizing exactly this kind of node status and scheduling reasoning.

4. Skip the Wait, Option 1 — Preventive Overprovisioning With PriorityClass

Even if node provisioning itself can't be made faster, you can arrange things ahead of time so you don't have to wait. That's the technique described in the official AWS containers blog.

Continuously deploy dummy Pods running nothing but a "pause" container, assigned a low-priority PriorityClass, to reserve spare capacity in the cluster. The moment a real, high-priority workload becomes Pending, the scheduler immediately preempts these dummy Pods and places the real Pod into the freed slot — no waiting on node provisioning, so the perceived delay is close to zero.

# Real workload
resources:
  requests:
    cpu: "1"
    memory: "500Mi"
priorityClassName: high-priority
---
# Dummy reserved capacity
resources:
  requests:
    cpu: "1"
    memory: "500Mi"
priorityClassName: low-priority

The same blog post also describes using a cluster-proportional-autoscaler to automatically adjust the number of dummy Pods based on cluster size (e.g., coresPerReplica: 8, nodesPerReplica: 4). That said, this approach assumes you're always leaving some spare capacity idle, and the resulting cost trade-off can't be ignored. How much spare capacity to carry depends on how often bursts occur and how much delay you can tolerate. As oneuptime's analysis points out, it's worth measuring each stage from demand spike to traffic being served (metric propagation, HPA update, Pod creation, node request, node boot, app readiness) to pinpoint the actual bottleneck before sizing your overprovisioning.

Managing this kind of spare capacity and its cost is also an area that tends to become tribal knowledge when handled manually. Kubo's AI-Driven Deployment can carry out this initial resource design from a natural-language instruction, freeing you from hand-assembling PriorityClass and dummy Pod YAML one piece at a time.

5. Skip the Wait, Option 2 — Moving From Cluster Autoscaler to Karpenter

Rather than trading cost for hidden delay through overprovisioning, there's also the option of making node provisioning itself faster. In an AWS environment, that means moving from Cluster Autoscaler to Karpenter.

Per CAST AI's 2026 comparison article, Cluster Autoscaler typically takes 3-4 minutes because it goes through a node-group abstraction layer (an ASG, for example), whereas Karpenter calls the cloud API directly (on AWS, EC2's RunInstances) and picks the best-fitting instance type from a broad candidate pool, getting a node Ready in 45-60 seconds. As Komodor's explainer also notes, Cluster Autoscaler is architecturally constrained by its 10-second polling interval, while Karpenter's event-driven design sidesteps that constraint entirely.

Which one to choose isn't a simple matter of "better vs. worse" — it depends on your operating model. If you're already running AWS EKS and burst traffic is frequent enough that the delay translates directly into business impact (SLA violations, degraded user experience), migrating to Karpenter is worth serious consideration. Conversely, if bursts are infrequent and your team already knows Cluster Autoscaler's behavior well, Option 1's overprovisioning is often good enough. In multi-cloud or on-prem environments running K3s, Karpenter isn't even on the table, so the realistic options are Option 1's design pattern, or improving the boot speed of whatever is actually provisioning nodes (VMware, Proxmox, bare metal).

Operating without being conscious of this layer distinction is a quietly draining task for infrastructure engineers. Kubo, as a managed Kubernetes built on K3s, is designed so that AI-Driven Deployment can handle this kind of node-provisioning judgment and execution directly — aiming to keep the full feature set of Kubernetes at roughly 58% of EKS's cost, while letting operators stop having to think about this "waiting time" at all.

6. Summary

When HPA "feels slow," in most cases it isn't HPA at all — it's a delay in a separate layer, node provisioning by Cluster Autoscaler. Three sequential stages — HPA's reaction time (1-3 minutes), Cluster Autoscaler's scan interval (10 seconds), and cloud-side node boot time (3-4 minutes) — combine into the roughly 5-minute delay you actually experience.

Think about the response in three stages: first diagnose by checking flags like --max-node-provision-time and scale.up.error.* error codes; then apply preventive overprovisioning with PriorityClass; and finally, if needed, make the more fundamental architectural change of migrating to Karpenter (in AWS environments). How far you go depends on how much business impact the delay is actually causing.

If you'd rather remove this kind of node-management complexity from your list of operational concerns entirely, a managed K3s service like Kubo is worth a look. If you'd like to talk through node-provisioning design, monitoring, and cost optimization, feel free to reach out via Contact Us.