1. The symptom — requests/limits are correct, but the Pod suddenly gets slow
requests and limits are set exactly as the manifest intends, yet one day a specific Pod starts responding slowly. It hasn't restarted. kubectl get events shows no OOMKilled. The node has plenty of memory headroom. Still, latency keeps getting worse. Has this ever happened to you?
A common assumption in K3s / Kubernetes operations is that "memory-related problems show up as OOMKill." Exceed the limit, get force-terminated, and it's recorded in the Pod events. So if there's no OOMKilled, many engineers conclude memory can't be the issue.
But in reality, there's a failure mode where the process is never killed, it just gets slow. The culprit is cgroup v2's memory.high — a throttling wall placed well before the limits boundary.
2. What we ruled out — memory leaks, GC tuning, node memory pressure. None of them
When facing this kind of symptom, infrastructure engineers usually check the usual suspects first.
- Application-level memory leak: a heap dump shows usage stable within requests
- Insufficient GC tuning: GC logs show no runaway full-GC cycles
- Node memory pressure (Node Pressure Eviction):
kubectl describe nodeshows no MemoryPressure condition
Ruling each one out still leaves the cause unexplained. At this point, the investigation needs to shift from the "application layer" to the "kernel / cgroup layer." Intermittent, hard-to-pin-down failures like this can easily eat up half a day of investigation on a self-managed cluster. With a managed environment like Kubo, that initial triage can often be shortened using operational know-how on the platform side.
3. The real cause — early throttling by cgroup v2's memory.high
The culprit turned out to be cgroup v2's memory.high, activated through a setting called memoryThrottlingFactor.
Kubernetes has a mechanism called Memory QoS that maps a Pod's requests/limits onto cgroup v2 memory control interfaces (KEP-2570: Memory QoS). Once enabled, it establishes a "throttling boundary" at the kernel level, one step before the limits line. That boundary is memory.high.
In this case, the root cause was that Memory QoS's memoryThrottlingFactor had been explicitly set, carried over from an older document's recommendation or a previous version's configuration. As a result, the application was being silently throttled well before memory usage actually reached the limits.
4. The four tiers of cgroup v2 memory control — how memory.max/min/low/high divide responsibilities
It's worth getting the cgroup v2 memory model straight. The Linux kernel's official documentation defines the memory controller through four interface files (Control Group v2 (Linux kernel official docs)).
- memory.min: as long as usage stays within this boundary, the cgroup's memory is never reclaimed under any condition — hard protection
- memory.low: a softer, best-effort protection than memory.min; it's only reclaimed if no other reclaimable memory exists
- memory.high: once exceeded, the process is throttled and placed under direct reclaim pressure. Critically, the OOM killer is not invoked at this point
- memory.max: the hard limit. Only when this is reached and memory can't be reduced does the OOM killer finally get invoked
In other words, monitoring only "limits = memory.max" isn't enough. The fact that memory.high is a separate layer that doesn't involve a kill is exactly what made this failure so hard to see.
Red Hat engineers also cite "better node stability under OOM pressure" as a benefit of cgroup v2 (cgroups v2 goes GA in OpenShift 4.13 (Red Hat official blog)). The early reclaim pressure from memory.high is designed to be a "gentler" behavior than an OOM Kill that takes down the whole node — but that gentleness can backfire by making the failure much harder to investigate.
5. Why there's no OOMKilled — the behavioral difference between memory.high and memory.max
When memory.max is exceeded, the kernel invokes the OOM killer and force-terminates the target process. That's the familiar "OOMKilled" recorded in Kubernetes Pod events.
When memory.high is exceeded, the behavior is completely different. The kernel doesn't kill the process; instead it triggers direct memory reclaim, evicting page cache and similar mechanisms to bring usage back down. That process itself takes time, which degrades application latency. But since the process keeps running, nothing is recorded in the Pod events.
This asymmetry is exactly what betrays the investigator's intuition that "no OOMKilled means memory isn't involved." Checking the memory.events file would show the high counter steadily increasing — but if you're only looking at Pod event logs, you'll never get there.
6. Diagnostic steps — checking memory.high/memory.current/memory.events in the cgroup files
The actual diagnosis starts not at the Pod level but by directly inspecting cgroup files on the node.
- Identify the Pod's node: run
kubectl get pod <pod-name> -o wideto find the node name - Locate the cgroup path: access the node and find the cgroup v2 path (under
/sys/fs/cgroup/kubepods.slice/...) from the container ID - Check the current memory.high value: if it isn't set to
max(unlimited), some configuration has put a throttling boundary in place - Check actual usage with memory.current: see whether actual usage is pinned right at the memory.high value
- Check the high counter in memory.events: if
highkeeps increasing, that's direct evidence throttling is occurring
Following these steps lets you prove, with objective numbers, that "there's no OOMKilled, but memory control is definitely involved." The underlying concept of cgroup v2-based resource isolation is also well explained in the official OpenShift/Kubernetes documentation.
7. The fix — reviewing memoryThrottlingFactor and your requests/limits design
There are broadly two directions for a fix.
First, reconsider memoryThrottlingFactor itself. The CNCF blog reports a real-world example: when this value is too aggressive, workloads like typical Java applications — which routinely run at high memory utilization — get throttled frequently enough to hurt application performance (How Katalyst guarantees memory QoS for colocated applications (CNCF blog)). Moving the value closer to 1.0 pushes the throttling boundary closer to memory.max, suppressing early activation.
Second, reconsider the requests/limits design itself. Since memory.high is set somewhere between requests and limits, workloads with a larger gap between the two are more exposed to early throttling. Deliberately narrowing that gap can be a more fundamental fix.
One thing worth stating precisely here: Memory QoS itself graduated to Beta in Kubernetes 1.37, with the feature gate enabled by default (Kubernetes v1.37: Memory QoS Graduates to Beta (official blog)). But that doesn't mean "upgrading to v1.37 alone starts throttling automatically." The official blog is explicit that "enabling a beta feature gate should not silently change application latency for workloads that previously ran without memory throttling," and the default value of memoryThrottlingFactor was changed to null. Unless you explicitly set this value, the kubelet will not write memory.high at all.
This kind of failure occurs when memoryThrottlingFactor has been explicitly set. In fact, the official documentation (Pod Quality of Service Classes) shows a KubeletConfiguration example with memoryThrottlingFactor: 0.9. Typical cases include following such official samples, older version documentation, a managed Kubernetes service's recommended settings, or simply carrying forward a configuration from a previous version (background details: Kubernetes v1.36: Tiered Memory Protection (official blog), Kubernetes 1.37 Memory QoS guide). Note that the exact formula and default coefficient differ across Kubernetes versions, so this article avoids asserting a single formula — when adopting or changing this setting, verify it empirically against your own cluster's kubelet logs and cgroup files.
Chasing down these kinds of settings on a self-managed K3s cluster, and keeping up with behavioral changes every version upgrade, is a non-trivial operational burden. A managed K3s offering like Kubo can track these default behavior changes on the platform side and keep configuration validity visible.
Summary
The assumption that "no OOMKilled means memory isn't involved" isn't necessarily true, because of cgroup v2's memory.high. Even with correctly configured requests/limits, if memoryThrottlingFactor has been explicitly set, throttling can kick in well before limits is ever reached.
Keep the diagnostic steps covered here in mind — locating the cgroup path, and checking memory.high / memory.current / memory.events — and the next time you run into an unexplained slowdown, you'll be equipped to widen the investigation beyond the application layer into the kernel and cgroup layer.
Keeping up with behavioral changes across Kubernetes version upgrades, down to cgroup v2-level configuration details, is no small burden for an infrastructure team. Using a managed K3s environment like Kubo On-Premise or Kubo is worth considering if you'd rather rethink that operational and investigative overhead altogether. If you're interested, feel free to reach out via Contact Us.