[{"data":1,"prerenderedAt":542},["ShallowReactive",2],{"blog-en-k3s-memory-qos-cgroup-v2-memory-high-throttling":3,"blog-related-en-k3s-memory-qos-cgroup-v2-memory-high-throttling":484,"blog-en-k3s-memory-qos-cgroup-v2-memory-high-throttling-alt":472},{"id":4,"title":5,"author":6,"body":7,"date":466,"description":467,"extension":468,"image":469,"locale":470,"meta":471,"navigation":472,"path":473,"seo":474,"stem":475,"tags":476,"__hash__":483},"blog\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling.md","requests and limits were correct, but my K3s Pod got slow anyway. The culprit wasn't OOMKill — it was cgroup v2's memory.high throttling early","Kubo Team",{"type":8,"value":9,"toc":454},"minimark",[10,15,23,31,34,46,50,56,59,85,96,100,106,116,131,141,145,151,160,193,200,209,213,219,225,237,248,252,258,261,309,312,316,322,325,340,346,369,401,408,412,418,424,436],[11,12,14],"h2",{"id":13},"_1-the-symptom-requestslimits-are-correct-but-the-pod-suddenly-gets-slow","1. The symptom — requests\u002Flimits are correct, but the Pod suddenly gets slow",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection01.webp",[16,24,25,26,30],{},"requests and limits are set exactly as the manifest intends, yet one day a specific Pod starts responding slowly. It hasn't restarted. ",[27,28,29],"code",{},"kubectl get events"," shows no OOMKilled. The node has plenty of memory headroom. Still, latency keeps getting worse. Has this ever happened to you?",[16,32,33],{},"A common assumption in K3s \u002F Kubernetes operations is that \"memory-related problems show up as OOMKill.\" Exceed the limit, get force-terminated, and it's recorded in the Pod events. So if there's no OOMKilled, many engineers conclude memory can't be the issue.",[16,35,36,37,41,42,45],{},"But in reality, there's a failure mode where the process is ",[38,39,40],"strong",{},"never killed, it just gets slow",". The culprit is cgroup v2's ",[27,43,44],{},"memory.high"," — a throttling wall placed well before the limits boundary.",[11,47,49],{"id":48},"_2-what-we-ruled-out-memory-leaks-gc-tuning-node-memory-pressure-none-of-them","2. What we ruled out — memory leaks, GC tuning, node memory pressure. None of them",[16,51,52],{},[19,53],{"alt":54,"src":55},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection02.webp",[16,57,58],{},"When facing this kind of symptom, infrastructure engineers usually check the usual suspects first.",[60,61,62,69,75],"ul",{},[63,64,65,68],"li",{},[38,66,67],{},"Application-level memory leak",": a heap dump shows usage stable within requests",[63,70,71,74],{},[38,72,73],{},"Insufficient GC tuning",": GC logs show no runaway full-GC cycles",[63,76,77,80,81,84],{},[38,78,79],{},"Node memory pressure (Node Pressure Eviction)",": ",[27,82,83],{},"kubectl describe node"," shows no MemoryPressure condition",[16,86,87,88,95],{},"Ruling each one out still leaves the cause unexplained. At this point, the investigation needs to shift from the \"application layer\" to the \"kernel \u002F cgroup layer.\" Intermittent, hard-to-pin-down failures like this can easily eat up half a day of investigation on a self-managed cluster. With a managed environment like ",[89,90,94],"a",{"href":91,"rel":92},"https:\u002F\u002Fkubo.hexabase.io\u002F",[93],"nofollow","Kubo",", that initial triage can often be shortened using operational know-how on the platform side.",[11,97,99],{"id":98},"_3-the-real-cause-early-throttling-by-cgroup-v2s-memoryhigh","3. The real cause — early throttling by cgroup v2's memory.high",[16,101,102],{},[19,103],{"alt":104,"src":105},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection03.webp",[16,107,108,109,111,112,115],{},"The culprit turned out to be cgroup v2's ",[27,110,44],{},", activated through a setting called ",[27,113,114],{},"memoryThrottlingFactor",".",[16,117,118,119,122,123,128,129,115],{},"Kubernetes has a mechanism called ",[38,120,121],{},"Memory QoS"," that maps a Pod's requests\u002Flimits onto cgroup v2 memory control interfaces (",[89,124,127],{"href":125,"rel":126},"https:\u002F\u002Fwww.kubernetes.dev\u002Fresources\u002Fkeps\u002F2570\u002F",[93],"KEP-2570: Memory QoS","). Once enabled, it establishes a \"throttling boundary\" at the kernel level, one step before the limits line. That boundary is ",[27,130,44],{},[16,132,133,134,136,137,140],{},"In this case, the root cause was that Memory QoS's ",[27,135,114],{}," had been ",[38,138,139],{},"explicitly set, carried over from an older document's recommendation or a previous version's configuration",". As a result, the application was being silently throttled well before memory usage actually reached the limits.",[11,142,144],{"id":143},"_4-the-four-tiers-of-cgroup-v2-memory-control-how-memorymaxminlowhigh-divide-responsibilities","4. The four tiers of cgroup v2 memory control — how memory.max\u002Fmin\u002Flow\u002Fhigh divide responsibilities",[16,146,147],{},[19,148],{"alt":149,"src":150},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection04.webp",[16,152,153,154,159],{},"It's worth getting the cgroup v2 memory model straight. The Linux kernel's official documentation defines the memory controller through four interface files (",[89,155,158],{"href":156,"rel":157},"https:\u002F\u002Fdocs.kernel.org\u002Fadmin-guide\u002Fcgroup-v2.html",[93],"Control Group v2 (Linux kernel official docs)",").",[60,161,162,168,174,187],{},[63,163,164,167],{},[38,165,166],{},"memory.min",": as long as usage stays within this boundary, the cgroup's memory is never reclaimed under any condition — hard protection",[63,169,170,173],{},[38,171,172],{},"memory.low",": a softer, best-effort protection than memory.min; it's only reclaimed if no other reclaimable memory exists",[63,175,176,178,179,182,183,186],{},[38,177,44],{},": once exceeded, the process is ",[38,180,181],{},"throttled and placed under direct reclaim pressure",". Critically, ",[38,184,185],{},"the OOM killer is not invoked"," at this point",[63,188,189,192],{},[38,190,191],{},"memory.max",": the hard limit. Only when this is reached and memory can't be reduced does the OOM killer finally get invoked",[16,194,195,196,199],{},"In other words, monitoring only \"limits = memory.max\" isn't enough. The fact that ",[38,197,198],{},"memory.high is a separate layer that doesn't involve a kill"," is exactly what made this failure so hard to see.",[16,201,202,203,208],{},"Red Hat engineers also cite \"better node stability under OOM pressure\" as a benefit of cgroup v2 (",[89,204,207],{"href":205,"rel":206},"https:\u002F\u002Fwww.redhat.com\u002Fja\u002Fblog\u002Fcgroups-v2-goes-ga-in-openshift-4.13",[93],"cgroups v2 goes GA in OpenShift 4.13 (Red Hat official blog)","). The early reclaim pressure from memory.high is designed to be a \"gentler\" behavior than an OOM Kill that takes down the whole node — but that gentleness can backfire by making the failure much harder to investigate.",[11,210,212],{"id":211},"_5-why-theres-no-oomkilled-the-behavioral-difference-between-memoryhigh-and-memorymax","5. Why there's no OOMKilled — the behavioral difference between memory.high and memory.max",[16,214,215],{},[19,216],{"alt":217,"src":218},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection05.webp",[16,220,221,222,224],{},"When ",[27,223,191],{}," is exceeded, the kernel invokes the OOM killer and force-terminates the target process. That's the familiar \"OOMKilled\" recorded in Kubernetes Pod events.",[16,226,221,227,229,230,233,234,115],{},[27,228,44],{}," is exceeded, the behavior is completely different. The kernel doesn't kill the process; instead it triggers ",[38,231,232],{},"direct memory reclaim",", evicting page cache and similar mechanisms to bring usage back down. That process itself takes time, which degrades application latency. But since the process keeps running, ",[38,235,236],{},"nothing is recorded in the Pod events",[16,238,239,240,243,244,247],{},"This asymmetry is exactly what betrays the investigator's intuition that \"no OOMKilled means memory isn't involved.\" Checking the ",[27,241,242],{},"memory.events"," file would show the ",[27,245,246],{},"high"," counter steadily increasing — but if you're only looking at Pod event logs, you'll never get there.",[11,249,251],{"id":250},"_6-diagnostic-steps-checking-memoryhighmemorycurrentmemoryevents-in-the-cgroup-files","6. Diagnostic steps — checking memory.high\u002Fmemory.current\u002Fmemory.events in the cgroup files",[16,253,254],{},[19,255],{"alt":256,"src":257},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection06.webp",[16,259,260],{},"The actual diagnosis starts not at the Pod level but by directly inspecting cgroup files on the node.",[262,263,264,274,284,294,300],"ol",{},[63,265,266,269,270,273],{},[38,267,268],{},"Identify the Pod's node",": run ",[27,271,272],{},"kubectl get pod \u003Cpod-name> -o wide"," to find the node name",[63,275,276,279,280,283],{},[38,277,278],{},"Locate the cgroup path",": access the node and find the cgroup v2 path (under ",[27,281,282],{},"\u002Fsys\u002Ffs\u002Fcgroup\u002Fkubepods.slice\u002F...",") from the container ID",[63,285,286,289,290,293],{},[38,287,288],{},"Check the current memory.high value",": if it isn't set to ",[27,291,292],{},"max"," (unlimited), some configuration has put a throttling boundary in place",[63,295,296,299],{},[38,297,298],{},"Check actual usage with memory.current",": see whether actual usage is pinned right at the memory.high value",[63,301,302,305,306,308],{},[38,303,304],{},"Check the high counter in memory.events",": if ",[27,307,246],{}," keeps increasing, that's direct evidence throttling is occurring",[16,310,311],{},"Following these steps lets you prove, with objective numbers, that \"there's no OOMKilled, but memory control is definitely involved.\" The underlying concept of cgroup v2-based resource isolation is also well explained in the official OpenShift\u002FKubernetes documentation.",[11,313,315],{"id":314},"_7-the-fix-reviewing-memorythrottlingfactor-and-your-requestslimits-design","7. The fix — reviewing memoryThrottlingFactor and your requests\u002Flimits design",[16,317,318],{},[19,319],{"alt":320,"src":321},"section07","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection07.webp",[16,323,324],{},"There are broadly two directions for a fix.",[16,326,327,333,334,339],{},[38,328,329,330,332],{},"First, reconsider ",[27,331,114],{}," itself."," The CNCF blog reports a real-world example: when this value is too aggressive, workloads like typical Java applications — which routinely run at high memory utilization — get throttled frequently enough to hurt application performance (",[89,335,338],{"href":336,"rel":337},"https:\u002F\u002Fwww.cncf.io\u002Fblog\u002F2024\u002F04\u002F25\u002Fhow-katalyst-guarantees-memory-qos-for-colocated-applications\u002F",[93],"How Katalyst guarantees memory QoS for colocated applications (CNCF blog)","). Moving the value closer to 1.0 pushes the throttling boundary closer to memory.max, suppressing early activation.",[16,341,342,345],{},[38,343,344],{},"Second, reconsider the requests\u002Flimits design itself."," Since memory.high is set somewhere between requests and limits, workloads with a larger gap between the two are more exposed to early throttling. Deliberately narrowing that gap can be a more fundamental fix.",[16,347,348,349,352,353,358,359,361,362,365,366,368],{},"One thing worth stating precisely here: ",[38,350,351],{},"Memory QoS itself graduated to Beta in Kubernetes 1.37, with the feature gate enabled by default"," (",[89,354,357],{"href":355,"rel":356},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2026\u002F09\u002F14\u002Fkubernetes-v1-37-memory-qos-graduates-to-beta\u002F",[93],"Kubernetes v1.37: Memory QoS Graduates to Beta (official blog)","). But that doesn't mean \"upgrading to v1.37 alone starts throttling automatically.\" The official blog is explicit that \"enabling a beta feature gate should not silently change application latency for workloads that previously ran without memory throttling,\" and the default value of ",[27,360,114],{}," was changed to ",[27,363,364],{},"null",". Unless you explicitly set this value, the kubelet will not write ",[27,367,44],{}," at all.",[16,370,371,372,374,375,378,379,384,385,388,389,394,395,400],{},"This kind of failure occurs when ",[27,373,114],{}," ",[38,376,377],{},"has been explicitly set",". In fact, the official documentation (",[89,380,383],{"href":381,"rel":382},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-qos\u002F",[93],"Pod Quality of Service Classes",") shows a KubeletConfiguration example with ",[27,386,387],{},"memoryThrottlingFactor: 0.9",". Typical cases include following such official samples, older version documentation, a managed Kubernetes service's recommended settings, or simply carrying forward a configuration from a previous version (background details: ",[89,390,393],{"href":391,"rel":392},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2026\u002F04\u002F29\u002Fkubernetes-v1-36-memory-qos-tiered-protection\u002F",[93],"Kubernetes v1.36: Tiered Memory Protection (official blog)",", ",[89,396,399],{"href":397,"rel":398},"https:\u002F\u002Fdevopsboys.com\u002Fblog\u002Fkubernetes-1-37-memory-qos-guide",[93],"Kubernetes 1.37 Memory QoS guide","). Note that the exact formula and default coefficient differ across Kubernetes versions, so this article avoids asserting a single formula — when adopting or changing this setting, verify it empirically against your own cluster's kubelet logs and cgroup files.",[16,402,403,404,407],{},"Chasing down these kinds of settings on a self-managed K3s cluster, and keeping up with behavioral changes every version upgrade, is a non-trivial operational burden. A managed K3s offering like ",[89,405,94],{"href":91,"rel":406},[93]," can track these default behavior changes on the platform side and keep configuration validity visible.",[11,409,411],{"id":410},"summary","Summary",[16,413,414],{},[19,415],{"alt":416,"src":417},"section08","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Fsection08.webp",[16,419,420,421,423],{},"The assumption that \"no OOMKilled means memory isn't involved\" isn't necessarily true, because of cgroup v2's memory.high. Even with correctly configured requests\u002Flimits, if ",[27,422,114],{}," has been explicitly set, throttling can kick in well before limits is ever reached.",[16,425,426,427,429,430,429,433,435],{},"Keep the diagnostic steps covered here in mind — locating the cgroup path, and checking ",[27,428,44],{}," \u002F ",[27,431,432],{},"memory.current",[27,434,242],{}," — and the next time you run into an unexplained slowdown, you'll be equipped to widen the investigation beyond the application layer into the kernel and cgroup layer.",[16,437,438,439,444,445,448,449,115],{},"Keeping up with behavioral changes across Kubernetes version upgrades, down to cgroup v2-level configuration details, is no small burden for an infrastructure team. Using a managed K3s environment like ",[89,440,443],{"href":441,"rel":442},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[93],"Kubo On-Premise"," or ",[89,446,94],{"href":91,"rel":447},[93]," is worth considering if you'd rather rethink that operational and investigative overhead altogether. If you're interested, feel free to reach out via ",[89,450,453],{"href":451,"rel":452},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[93],"Contact Us",{"title":455,"searchDepth":456,"depth":456,"links":457},"",2,[458,459,460,461,462,463,464,465],{"id":13,"depth":456,"text":14},{"id":48,"depth":456,"text":49},{"id":98,"depth":456,"text":99},{"id":143,"depth":456,"text":144},{"id":211,"depth":456,"text":212},{"id":250,"depth":456,"text":251},{"id":314,"depth":456,"text":315},{"id":410,"depth":456,"text":411},"2026-11-06","A K3s Pod with correctly configured requests\u002Flimits suddenly got slow, with no OOMKilled event anywhere. The real cause was cgroup v2's memory.high throttling. Here's how to diagnose and fix it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling",{"title":5,"description":467},"blog\u002Fen\u002Fk3s-memory-qos-cgroup-v2-memory-high-throttling",[477,478,479,480,481,482],"k3s","kubernetes","cgroup-v2","memory-qos","memory-high","resource-management","lS2OKJMyzvbBMOJDBBAr3PA_F6FCt0ZhCGLbjIig660",[485,495,503,512,522,532],{"path":486,"title":487,"description":488,"date":489,"tags":490},"\u002Fblog\u002Fen\u002Fk3s-gpu-oomkilled-vpa-resource-estimation","GPU Memory Ran Out and Killed an AI Training Job. Why VPA Can't Save You on K3s","Why AI training jobs on K3s get OOMKilled from GPU memory shortages, why Vertical Pod Autoscaler doesn't cover GPU resources, and how to size requests\u002Flimits correctly using measurement-based design.","2026-09-20",[477,478,491,492,493,494,482],"gpu","oomkilled","vpa","ai-infrastructure",{"path":496,"title":497,"description":498,"date":499,"tags":500},"\u002Fblog\u002Fen\u002Fai-generated-kubernetes-manifest-resource-overprovisioning","Kubernetes Resource Design Can't Be Left to AI: Why 'Working' YAML Is Wasting 69% of Your Cloud Bill","AI-generated Kubernetes manifests pass kubectl apply and 'work' — but getting Kubernetes resource design wrong drives massive overprovisioning. Here's why AI struggles with production-grade requests\u002Flimits and what to check before you ship.","2026-08-04",[477,478,482,501,502],"capacity-planning","managed-kubernetes",{"path":504,"title":505,"description":506,"date":507,"tags":508},"\u002Fblog\u002Fen\u002Fai-manifest-generation-kubernetes-architecture-bottleneck","AI Can Write a YAML File in One Second, But Your Cluster Won't Get Any Faster. Why the Real Bottleneck in Kubernetes Operations Is Architecture, Not Code","Generating Kubernetes manifests at AI speed doesn't make production Kubernetes operations faster. This article breaks down three real bottlenecks — CPU throttling, the HPA\u002FVPA conflict, and database connection starvation — and how to split the work between AI and humans.","2026-07-18",[478,477,482,509,510,511],"autoscaling","ai-ops","devops",{"path":513,"title":514,"description":515,"date":516,"tags":517},"\u002Fblog\u002Fen\u002Fk3s-ai-agent-audit-log-traceability-gap","We Couldn't Reconstruct What an AI Agent Did to Our Kubernetes API. The Gap Was in Our Audit Log Settings","After running AI agents on Kubernetes, we realized that when something went wrong, the Audit Log alone couldn't tell us who did what, or why. Here's the Audit Policy blind spot and how we fixed agent traceability.","2026-11-09",[477,478,518,519,520,521],"ai-agent","audit-log","rbac","observability",{"path":523,"title":524,"description":525,"date":526,"tags":527},"\u002Fblog\u002Fen\u002Fk3s-kubelet-eviction-inode-exhaustion-diskpressure","K3s Pods Kept Getting Evicted. Disk Space Was Fine — Inodes Were Not","K3s pods kept getting evicted while df -h looked perfectly healthy. The real cause wasn't disk space but inode exhaustion. We cover kubelet's eviction thresholds, how to diagnose with df -i, and how to prevent it.","2026-11-08",[477,478,528,529,530,531],"kubelet","eviction","diskpressure","troubleshooting",{"path":533,"title":534,"description":535,"date":536,"tags":537},"\u002Fblog\u002Fen\u002Fk3s-cilium-kube-proxy-replacement-nodeport-unreachable","We Swapped kube-proxy for Cilium on K3s, and NodePort Stopped Working","We followed the official Cilium guide to enable kube-proxy replacement on K3s, and NodePort traffic suddenly stopped reaching our pods. The cause: K3s-specific defaults (Flannel, ServiceLB) that the generic docs never mention. Here's the fix and a pre-production checklist.","2026-11-07",[477,478,538,539,540,541],"cilium","ebpf","kube-proxy","networking",1794305515073]