[{"data":1,"prerenderedAt":505},["ShallowReactive",2],{"blog-en-k3s-cronjob-silent-failure-concurrencypolicy-missed-schedule":3,"blog-related-en-k3s-cronjob-silent-failure-concurrencypolicy-missed-schedule":453,"blog-en-k3s-cronjob-silent-failure-concurrencypolicy-missed-schedule-alt":441},{"id":4,"title":5,"author":6,"body":7,"date":435,"description":436,"extension":437,"image":438,"locale":439,"meta":440,"navigation":441,"path":442,"seo":443,"stem":444,"tags":445,"__hash__":452},"blog\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule.md","The CronJob Stayed Green. It Hadn't Run in 24 Days. 4 Traps Behind K3s's Silent Scheduling Failures","Kubo Team",{"type":8,"value":9,"toc":423},"minimark",[10,19,35,38,43,58,65,74,82,94,100,125,148,154,161,166,172,181,193,205,209,222,228,238,250,261,267,271,274,280,293,308,311,315,327,333,367,375,382,386,394,401,414],[11,12,13,14,18],"p",{},"One morning, an engineer checked the last successful run time of a backup job and went pale. The CronJob's status had been showing \"Succeeded\" the whole time. In reality, that backup ",[15,16,17],"strong",{},"hadn't run once in 24 days",". Under standard monitoring, this kind of failure looks completely healthy.",[11,20,21,22,26,27,30,31,34],{},"K3s and Kubernetes CronJobs look simple as a scheduling mechanism. But when four factors line up in a specific combination — ",[23,24,25],"code",{},"concurrencyPolicy",", ",[23,28,29],{},"startingDeadlineSeconds",", the 100-missed-schedule limit, and what exit code semantics actually mean — they can produce a nasty failure mode where ",[15,32,33],{},"the status stays healthy while execution quietly stops",". Each piece behaves exactly as documented on its own; it's the combination that turns into silence nobody notices.",[11,36,37],{},"This article breaks down, based on a reported real-world incident, four traps that make CronJobs on K3s stop silently, then explains why Kubernetes-native monitoring alone isn't enough and how to turn it into monitoring that actually notices.",[39,40,42],"h2",{"id":41},"_1-the-cronjob-stayed-green-while-the-backup-sat-idle-for-24-days","1. The CronJob Stayed Green While the Backup Sat Idle for 24 Days",[11,44,45,46,49,50,57],{},"In one reported case, a CronJob running a nightly backup went ",[15,47,48],{},"24 days without a single execution",". The CronJob's own status kept showing \"Succeeded,\" with no outward sign of trouble. What finally surfaced the problem was someone manually checking the last successful run timestamp (",[51,52,56],"a",{"href":53,"rel":54},"https:\u002F\u002Fdev.to\u002Fkrissv\u002Fkubernetes-cronjobs-silently-fail-more-than-you-think-2nb9",[55],"nofollow","analysis on dev.to",").",[11,59,60],{},[61,62],"img",{"alt":63,"src":64},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Fsection01.webp",[11,66,67,68,73],{},"What this case shows is that a CronJob controller won't necessarily self-report when something is wrong. Even when the Kubernetes cluster itself is perfectly healthy, a single CronJob can be quietly stalled. Whether you catch this kind of silent failure often comes down to whether CronJob monitoring was built in from day one (",[51,69,72],{"href":70,"rel":71},"https:\u002F\u002Fkubo.hexabase.io\u002F",[55],"Kubo Cloud"," is one example of a K3s environment with a monitoring stack included by default — that's part of what makes the difference). The following sections break down, based on the official spec, exactly what combination of settings produces this silence.",[39,75,77,78,81],{"id":76},"_2-trap-1-concurrencypolicy-forbid-chooses-to-skip-quietly","2. Trap 1: ",[23,79,80],{},"concurrencyPolicy: Forbid"," Chooses to Skip Quietly",[11,83,84,85,87,88,93],{},"A CronJob's ",[23,86,25],{}," has three options. According to the ",[51,89,92],{"href":90,"rel":91},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fcron-jobs",[55],"official Kubernetes documentation",", each behaves as follows:",[11,95,96],{},[61,97],{"alt":98,"src":99},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Fsection02.webp",[101,102,103,110,119],"ul",{},[104,105,106,109],"li",{},[15,107,108],{},"Allow"," (default): starts a new Job even if the previous one is still running",[104,111,112,115,116],{},[15,113,114],{},"Forbid",": if the previous Job hasn't finished, ",[15,117,118],{},"skips starting the new Job entirely",[104,120,121,124],{},[15,122,123],{},"Replace",": cancels the running Job and replaces it with the new one",[11,126,127,128,130,131,136,137,139,140,143,144,147],{},"Many teams choose ",[23,129,114],{}," for work like backups or data syncs, where a double-run would cause real problems. In fact, a ",[51,132,135],{"href":133,"rel":134},"https:\u002F\u002Ftechblog.zozo.com\u002Fentry\u002Fkubernetes-cronjob-monitoring",[55],"ZOZO TECH BLOG write-up on CronJob operations"," shows a configuration example that pairs ",[23,138,25],{}," with ",[23,141,142],{},"activeDeadlineSeconds"," and ",[23,145,146],{},"backoffLimit"," for monitored CronJobs.",[11,149,150,151,153],{},"The catch is that the \"skip\" ",[23,152,114],{}," chooses is a quiet one — it never creates a Job object in the first place. As long as the prior Job keeps running, every schedule in between disappears without a trace. That's where the next trap compounds the problem.",[39,155,157,158,160],{"id":156},"_3-trap-2-startingdeadlineseconds-doesnt-always-protect-you","3. Trap 2: ",[23,159,29],{}," Doesn't Always Protect You",[11,162,163,165],{},[23,164,29],{}," defines how many seconds a Job can start late before Kubernetes gives up on it. The official documentation states it plainly:",[11,167,168],{},[61,169],{"alt":170,"src":171},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Fsection03.webp",[173,174,175],"blockquote",{},[11,176,177,178,180],{},"If a Job fails to start by its scheduled time and more time than ",[23,179,29],{}," has passed, that run is skipped. If left unset, there is no deadline.",[11,182,183,184,186,187,189,190],{},"This looks straightforward on its own, but combined with ",[23,185,80],{}," the behavior gets more complicated. While the previous Job is still running, new runs keep getting skipped; once it finally finishes, ",[23,188,29],{}," is re-evaluated at that moment. ",[15,191,192],{},"If the deadline has already passed by then, the run that was waiting simply vanishes.",[11,194,195,196,201,202,204],{},"A ",[51,197,200],{"href":198,"rel":199},"https:\u002F\u002Fengineering.mercari.com\u002Fblog\u002Fentry\u002Fk8s-cronjob-20200908\u002F",[55],"Mercari Engineering blog post"," argues that ",[23,203,29],{}," should be treated as \"practically mandatory.\" Leaving it unset during an extended outage or maintenance window leads directly into the next trap.",[39,206,208],{"id":207},"_4-trap-3-the-100-missed-schedule-limit-and-clock-skew","4. Trap 3: The 100-Missed-Schedule Limit and Clock Skew",[11,210,211,212,215,216,221],{},"The CronJob controller has a built-in rule: ",[15,213,214],{},"after 100 consecutive missed schedules, it stops generating Jobs altogether."," ",[51,217,220],{"href":218,"rel":219},"https:\u002F\u002Fplatform9.com\u002Fkb\u002Fpmk\u002Fsolutions\u002Fkubernetes-cronjobs-failed-with-error-too-many-missed-start-time",[55],"Platform9's knowledge base"," documents the actual error message this produces.",[11,223,224],{},[61,225],{"alt":226,"src":227},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Fsection04.webp",[229,230,235],"pre",{"className":231,"code":233,"language":234},[232],"language-text","Cannot determine if job needs to be started: too many missed start time (> 100).\nSet or decrease .spec.startingDeadlineSeconds or check clock skew\n","text",[23,236,233],{"__ignoreMap":237},"",[11,239,240,241,245,246,249],{},"According to ",[51,242,244],{"href":198,"rel":243},[55],"Mercari Engineering's explanation",", reaching this limit doesn't always mean 100 genuine failures occurred. Simply leaving a CronJob ",[23,247,248],{},"suspend: true"," for a few days can accumulate enough missed schedules during that window to hit the same threshold.",[11,251,252,253,256,257,260],{},"Platform9's write-up adds another wrinkle: if ",[23,254,255],{},"status.lastScheduleTime"," is unset or null, the controller falls back to the much older ",[23,258,259],{},"metadata.creationTimestamp"," to calculate missed schedules, which can push an otherwise healthy CronJob over the 100-miss threshold by mistake. Clock skew between nodes can trigger the same false positive.",[11,262,263,264],{},"Once a CronJob is in this state, a simple patch usually isn't enough to recover it. Both write-ups recommend ",[15,265,266],{},"recreating the CronJob resource itself.",[39,268,270],{"id":269},"_5-trap-4-exit-code-0-doesnt-guarantee-any-real-work-happened","5. Trap 4: Exit Code 0 Doesn't Guarantee Any Real Work Happened",[11,272,273],{},"The three traps so far all involved a Job never starting in the first place. But even once a Job does start, you're not in the clear yet.",[11,275,276],{},[61,277],{"alt":278,"src":279},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Fsection05.webp",[11,281,282,283,288,289,292],{},"As the ",[51,284,287],{"href":285,"rel":286},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fjob\u002F",[55],"official Kubernetes documentation for Jobs"," explains, Kubernetes follows Unix convention: ",[15,290,291],{},"if a container exits with code 0, that alone is treated as success."," Whether the workload actually had anything to process, or processed the expected volume of data, is not something Kubernetes checks.",[11,294,295,299,300,303,304,307],{},[51,296,298],{"href":53,"rel":297},[55],"The dev.to analysis"," points out that this lets a Job that \"processed an empty queue\" or \"wrote a zero-byte backup\" get marked as successful all the same. On top of that, automatic garbage collection driven by ",[23,301,302],{},"successfulJobsHistoryLimit"," (default 3) and ",[23,305,306],{},"failedJobsHistoryLimit"," (default 1) means the logs and history you'd need to investigate disappear quickly too.",[11,309,310],{},"In other words, there's a fourth kind of silence here: \"the Job did start, but effectively did nothing.\"",[39,312,314],{"id":313},"_6-turning-this-into-monitoring-that-actually-notices","6. Turning This Into Monitoring That Actually Notices",[11,316,317,318,215,321,326],{},"What all four traps share is that ",[15,319,320],{},"Kubernetes' internal mechanics never self-report the anomaly.",[51,322,325],{"href":323,"rel":324},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-02-09-monitor-cronjob-missed-schedules\u002Fview",[55],"OneUptime's monitoring guide"," lays out the following approach:",[11,328,329],{},[61,330],{"alt":331,"src":332},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Fsection06.webp",[101,334,335,347,357,364],{},[104,336,337,338,26,340,343,344],{},"Regularly check the CronJob's ",[23,339,255],{},[23,341,342],{},"status.lastSuccessfulTime",", and ",[23,345,346],{},"status.active",[104,348,349,350,143,353,356],{},"Collect ",[23,351,352],{},"kube_cronjob_status_last_schedule_time",[23,354,355],{},"kube_cronjob_status_last_successful_time",", exposed by kube-state-metrics, through Prometheus",[104,358,359,360,363],{},"Set alert thresholds at roughly ",[15,361,362],{},"twice the expected interval"," (around 48 hours for a daily job, around 2 hours for an hourly one)",[104,365,366],{},"Pair this with external, cluster-independent monitoring (a dead man's switch) that can still detect an outage even when the cluster itself goes quiet",[11,368,369,370,374],{},"ZOZO TECH BLOG's case shows the same principle in practice: rather than relying solely on Kubernetes-native mechanisms, they built a threshold-based alerting setup on top of an external dashboard (",[51,371,373],{"href":133,"rel":372},[55],"same article","). Given that the in-cluster controller can go silent, having a way to check \"when did this last actually succeed?\" from outside the cluster is the common defense against all four traps above.",[11,376,377,378,381],{},"Building this kind of monitoring stack from scratch — Prometheus and Grafana alone — takes real engineering effort. ",[51,379,72],{"href":70,"rel":380},[55]," is a managed Kubernetes platform built on K3s that ships with Prometheus and Grafana included by default, so you don't have to build a CronJob status dashboard from zero.",[39,383,385],{"id":384},"summary","Summary",[11,387,388,389,26,391,393],{},"CronJobs don't go silent because of a single misconfiguration — it's ",[23,390,25],{},[23,392,29],{},", the 100-missed-schedule limit, and exit code 0 semantics combining into a silence that's hard to notice precisely because each piece behaves exactly as documented.",[11,395,396,397,400],{},"The real fix is to stop assuming \"Kubernetes will tell me when something's wrong,\" and instead build in, from the start, external monitoring of status fields like ",[23,398,399],{},"lastScheduleTime"," and the metrics around them.",[11,402,403,404,407,408,413],{},"How much of that monitoring your team builds and maintains yourselves, versus how much you get from a platform that ships with it, is a question of where you want to spend engineering time. ",[51,405,72],{"href":70,"rel":406},[55]," launches K3s clusters with Prometheus and Grafana already wired up, so the monitoring design covered in this article doesn't have to start from a blank slate. Take a look at the ",[51,409,412],{"href":410,"rel":411},"https:\u002F\u002Fwww.hexabase.com\u002Fpricing\u002F",[55],"pricing plans"," and compare it against what your current CronJob operations actually cost you.",[11,415,416,417,422],{},"If you're running CronJobs on-premises or in a closed network where even setting up external monitoring is its own challenge, ",[51,418,421],{"href":419,"rel":420},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[55],"Kubo On-Premise"," is worth considering — it lets you build the same kind of monitoring design without sending data outside your own environment.",{"title":237,"searchDepth":424,"depth":424,"links":425},2,[426,427,429,431,432,433,434],{"id":41,"depth":424,"text":42},{"id":76,"depth":424,"text":428},"2. Trap 1: concurrencyPolicy: Forbid Chooses to Skip Quietly",{"id":156,"depth":424,"text":430},"3. Trap 2: startingDeadlineSeconds Doesn't Always Protect You",{"id":207,"depth":424,"text":208},{"id":269,"depth":424,"text":270},{"id":313,"depth":424,"text":314},{"id":384,"depth":424,"text":385},"2026-09-16","K3s and Kubernetes CronJobs can keep showing 'Succeeded' while quietly not running for weeks. Here's how concurrencyPolicy, startingDeadlineSeconds, the 100-missed-schedule limit, and exit code 0 combine to create silent failures, and how to monitor for it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule",{"title":5,"description":436},"blog\u002Fen\u002Fk3s-cronjob-silent-failure-concurrencypolicy-missed-schedule",[446,447,448,449,450,451],"k3s","kubernetes","cronjob","concurrencypolicy","monitoring","sre","s2XNXhfhFdpD_putrK7IjmL-IUiRSFO-EZtTnyHAeoE",[454,463,472,480,488,495],{"path":455,"title":456,"description":457,"date":458,"tags":459},"\u002Fblog\u002Fen\u002Fk3s-litmuschaos-chaos-engineering-blast-radius","Deliberately Breaking Your Own Production K3s: How LitmusChaos Chaos Engineering Helps You Choose the Right First Workload to Break","You can't know how resilient your production K3s environment really is until something actually breaks. Monitoring and alerting only detect failure after the fact. This guide covers how to get started with chaos engineering using CNCF's LitmusChaos on K3s, and how to judge which workloads are safe to break first and which ones demand extra caution.","2026-09-04",[446,447,460,461,462,451],"chaos-engineering","litmuschaos","resilience",{"path":464,"title":465,"description":466,"date":467,"tags":468},"\u002Fblog\u002Fen\u002Fk3s-servicemonitor-prometheus-not-scraping-connection-points","Your ServiceMonitor YAML Is Correct. So Why Isn't Prometheus Scraping It? A K3s Diagnostic Guide to the 'Connection Points'","You configured a kubernetes service monitor on K3s, but nothing shows up in Prometheus Targets. 90% of the time, the cause is one of four connection points: label selectors, port names, RBAC, or the namespaceSelector. Here's a step-by-step diagnostic guide.","2026-08-30",[446,447,469,470,450,471],"prometheus","service-monitor","observability",{"path":473,"title":474,"description":475,"date":476,"tags":477},"\u002Fblog\u002Fen\u002Fkubernetes-high-availability-broadcast-seamless-switching","Broadcasters Send the Same Video Twice and Just Keep Whichever Arrives First — It Turns Out That's Exactly How Kubernetes Achieves High Availability","World Cup broadcasts duplicate every camera feed across two independent paths and simply discard whichever packet arrives second. That seemingly wasteful design shares the same philosophy as Kubernetes high availability and multi-AZ architecture.","2026-08-05",[446,447,478,479,451],"high-availability","multi-az",{"path":481,"title":482,"description":483,"date":484,"tags":485},"\u002Fblog\u002Fen\u002Fedge-k3s-observability-homelab-dashboard","A Wall-Mounted Dashboard Taught Me What 'Peace of Mind' Really Means: Observability Design Lessons for Edge K3s Clusters","The trial-and-error troubleshooting behind a home-lab wall-mounted dashboard is the same trap that hits edge Kubernetes clusters scattered across factories and stores. Drawing on official K3s, Prometheus, and Grafana docs, this piece lays out design principles for not deferring observability.","2026-07-26",[446,447,486,471,450,487],"edge-computing","managed-kubernetes",{"path":489,"title":490,"description":491,"date":492,"tags":493},"\u002Fblog\u002Fen\u002Fbroadcast-redundancy-kubernetes-high-availability-design","Why Does the FIFA World Cup Broadcast Never Go Dark? What Broadcast Engineering's Dual-Path Redundancy Teaches Us About Real Kubernetes High Availability","Kubernetes high availability isn't achieved simply by adding more replicas. Using the broadcast industry's SMPTE ST 2022-7 dual-path transmission as a lens, this article unpacks what Topology Spread Constraints and multi-AZ design actually mean.","2026-07-20",[447,446,478,494,451,487],"topology-spread-constraints",{"path":496,"title":497,"description":498,"date":499,"tags":500},"\u002Fblog\u002Fen\u002Fk3s-hpa-containerresource-sidecar-utilization-dilution","HPA Was Scaling on the Sidecar's Share Too. The ContainerResource Blind Spot in Kubernetes","Kubernetes HPA sums resource usage across every container in a Pod, so Istio sidecars and logging agents dilute the scaling decision. Here's how the ContainerResource metric fixes it, in a K3s\u002FKubo context.","2026-10-21",[446,447,501,502,503,504],"hpa","containerresource","autoscaling","service-mesh",1792592633126]