"CPU Usage Is Low, But It's Slow" — The Classic Way Teams Misdiagnose Kubernetes Latency
You open the production dashboard and CPU usage sits at around 30%. Yet API response times are clearly slow. This "contradictory dashboard" is the first trap that leads many infrastructure engineers to misdiagnose the true cause of Kubernetes latency.
The root cause lies in how CPU limits are enforced at the kernel level. According to the official Kubernetes documentation, CPU limits are enforced by the CFS (Completely Fair Scheduler), and once a container approaches its limit, the kernel restricts its access to the CPU itself. Unlike memory, where a process gets killed by OOM, there's no obvious error — just a vague sense that things are "slow."
What makes this worse is that the "CPU usage" shown by most monitoring tools is measured after throttling has already occurred. As Datadog's blog post points out, a container's usage looks low precisely because it's being restricted — the intuition that "low usage means there's headroom" simply doesn't hold here. IBM's troubleshooting guide similarly notes that investigations focused only on CPU usage tend to miss throttling entirely.
Investigating this kind of "latency that contradicts what the dashboard shows" takes a long time if your monitoring stack isn't already in place. With a managed K3s environment like Kubo, which comes with Prometheus and Grafana built in, you don't need to spend extra time building out this kind of observability from scratch.
Why Adding More Pods (Scaling with HPA) Can Make Things Worse
Suspecting CPU throttling, you scale out by adding more replicas — but latency doesn't improve, or even gets worse. This usually means the bottleneck actually lives in the application layer or the database layer.
A classic example is database connection pool exhaustion. PostgreSQL manages the number of concurrent connections via the max_connections parameter, and as PlanetScale explains, Postgres forks an OS process for every connection, so memory usage and context-switching costs rise sharply as connections grow. When HPA adds more Pods on top of this, and each Pod maintains its own connection pool, the total number of connection requests balloons even further.
CubeAPM's technical article walks through exactly this pattern of connection exhaustion in Kubernetes-based applications. Adding Pods is supposed to increase capacity — but against a database's connection limit, it backfires, triggering a surge of "too many clients" errors instead.
A well-known way to avoid this is a connection pooler. As Neon's official documentation explains, placing a pooler like PgBouncer in between lets you multiplex many application-level connections down to a small number of real database connections, allowing Pods to scale horizontally without ever hitting the max_connections ceiling.
The Metric You Should Actually Be Watching Isn't CPU% — How to Isolate the Real Cause of Kubernetes Latency
If you only ever look at CPU usage, you risk missing both throttling and connection pool exhaustion at once. Last9's engineering blog explains that the Prometheus metric container_cpu_cfs_throttled_seconds_total — not CPU usage — directly shows how much time was actually spent throttled. Even when CPU usage looks low, a high value here is a red flag for throttling.
On the database side, isolating latency requires visibility into connection pool utilization and queue wait times. If HPA is configured to scale on CPU alone, Google Cloud's official documentation shows that since Kubernetes 1.6, the Custom Metrics API lets you feed application-specific metrics — like queue length or response time — into HPA's scaling conditions via Prometheus. That's the first step away from a setup that scales purely on CPU.
In short, correctly identifying the true cause of Kubernetes latency requires looking at a minimum of four metrics side by side: CPU%, CFS throttled time, database connection pool utilization, and p99 latency. Relying on just one of these will always leave you blind to at least one other possible cause.
The Right Fix: What to Do Before You Add More Pods
Consider Removing CPU Limits Entirely for Latency-Sensitive Services
groundcover's analysis points out that setting CPU limits too aggressively can cause containers to be throttled unnecessarily even when the node has spare capacity. Rather than applying a CPU limit to every workload, it's worth considering securing priority via requests while relaxing or removing limits for latency-sensitive services.
Switch HPA's Scaling Trigger from CPU to Application-Level Metrics
Using the Custom Metrics API described above, switching from CPU-based scaling to scaling on request queue length or response time lets you catch the "CPU is free but things are backed up" state much earlier.
Put a Connection Pooler in the Middle
As Komodor's article also points out, misdiagnosing resource issues consumes a huge amount of operational time. Placing a pooler like PgBouncer as a sidecar or intermediary layer decouples Pod count growth from database connection count growth.
Size maxReplicas Backwards from What the Database Can Actually Accept
Instead of setting HPA's maxReplicas based solely on cluster CPU capacity, calculate it backwards from "the number of connections the database can safely accept ÷ the number of connections per Pod." This prevents scale-out from becoming a self-inflicted source of errors.
Having all of these metrics visible on a single screen from day one is the fastest way to avoid a drawn-out misdiagnosis. Kubo Cloud is a managed Kubernetes service built on K3s with Prometheus and Grafana included by default, so you can start tracking CFS throttled time and setting up custom-metric-based HPA immediately, without building an observability stack from scratch.
Conclusion: Scaling Isn't Magic
"Add more Pods" is a convenient fix, but it isn't a cure-all. When the real bottleneck is kernel-level CPU throttling or database-side connection pool exhaustion, scaling out doesn't just fail to fix the problem — it can actively introduce a new one in the form of connection errors.
Correctly identifying the cause of Kubernetes latency starts with not trusting a single CPU usage number, and instead making throttled time, database connection pool utilization, and p99 latency all visible at once.
Building an observability stack from scratch on EKS or AKS costs time and money. With Kubo, a K3s-based service with Prometheus and Grafana included starting at ¥48,000/month, you can start using this kind of visibility right away. If you'd like to try running standard, vendor-lock-in-free Kubernetes without losing time to misdiagnosis, get in touch with us to learn more.