One morning, an engineer checked the last successful run time of a backup job and went pale. The CronJob's status had been showing "Succeeded" the whole time. In reality, that backup hadn't run once in 24 days. Under standard monitoring, this kind of failure looks completely healthy.
K3s and Kubernetes CronJobs look simple as a scheduling mechanism. But when four factors line up in a specific combination — concurrencyPolicy, startingDeadlineSeconds, the 100-missed-schedule limit, and what exit code semantics actually mean — they can produce a nasty failure mode where the status stays healthy while execution quietly stops. Each piece behaves exactly as documented on its own; it's the combination that turns into silence nobody notices.
This article breaks down, based on a reported real-world incident, four traps that make CronJobs on K3s stop silently, then explains why Kubernetes-native monitoring alone isn't enough and how to turn it into monitoring that actually notices.
1. The CronJob Stayed Green While the Backup Sat Idle for 24 Days
In one reported case, a CronJob running a nightly backup went 24 days without a single execution. The CronJob's own status kept showing "Succeeded," with no outward sign of trouble. What finally surfaced the problem was someone manually checking the last successful run timestamp (analysis on dev.to).
What this case shows is that a CronJob controller won't necessarily self-report when something is wrong. Even when the Kubernetes cluster itself is perfectly healthy, a single CronJob can be quietly stalled. Whether you catch this kind of silent failure often comes down to whether CronJob monitoring was built in from day one (Kubo Cloud is one example of a K3s environment with a monitoring stack included by default — that's part of what makes the difference). The following sections break down, based on the official spec, exactly what combination of settings produces this silence.
2. Trap 1: concurrencyPolicy: Forbid Chooses to Skip Quietly
A CronJob's concurrencyPolicy has three options. According to the official Kubernetes documentation, each behaves as follows:
- Allow (default): starts a new Job even if the previous one is still running
- Forbid: if the previous Job hasn't finished, skips starting the new Job entirely
- Replace: cancels the running Job and replaces it with the new one
Many teams choose Forbid for work like backups or data syncs, where a double-run would cause real problems. In fact, a ZOZO TECH BLOG write-up on CronJob operations shows a configuration example that pairs concurrencyPolicy with activeDeadlineSeconds and backoffLimit for monitored CronJobs.
The catch is that the "skip" Forbid chooses is a quiet one — it never creates a Job object in the first place. As long as the prior Job keeps running, every schedule in between disappears without a trace. That's where the next trap compounds the problem.
3. Trap 2: startingDeadlineSeconds Doesn't Always Protect You
startingDeadlineSeconds defines how many seconds a Job can start late before Kubernetes gives up on it. The official documentation states it plainly:
If a Job fails to start by its scheduled time and more time than
startingDeadlineSecondshas passed, that run is skipped. If left unset, there is no deadline.
This looks straightforward on its own, but combined with concurrencyPolicy: Forbid the behavior gets more complicated. While the previous Job is still running, new runs keep getting skipped; once it finally finishes, startingDeadlineSeconds is re-evaluated at that moment. If the deadline has already passed by then, the run that was waiting simply vanishes.
A Mercari Engineering blog post argues that startingDeadlineSeconds should be treated as "practically mandatory." Leaving it unset during an extended outage or maintenance window leads directly into the next trap.
4. Trap 3: The 100-Missed-Schedule Limit and Clock Skew
The CronJob controller has a built-in rule: after 100 consecutive missed schedules, it stops generating Jobs altogether. Platform9's knowledge base documents the actual error message this produces.
Cannot determine if job needs to be started: too many missed start time (> 100).
Set or decrease .spec.startingDeadlineSeconds or check clock skew
According to Mercari Engineering's explanation, reaching this limit doesn't always mean 100 genuine failures occurred. Simply leaving a CronJob suspend: true for a few days can accumulate enough missed schedules during that window to hit the same threshold.
Platform9's write-up adds another wrinkle: if status.lastScheduleTime is unset or null, the controller falls back to the much older metadata.creationTimestamp to calculate missed schedules, which can push an otherwise healthy CronJob over the 100-miss threshold by mistake. Clock skew between nodes can trigger the same false positive.
Once a CronJob is in this state, a simple patch usually isn't enough to recover it. Both write-ups recommend recreating the CronJob resource itself.
5. Trap 4: Exit Code 0 Doesn't Guarantee Any Real Work Happened
The three traps so far all involved a Job never starting in the first place. But even once a Job does start, you're not in the clear yet.
As the official Kubernetes documentation for Jobs explains, Kubernetes follows Unix convention: if a container exits with code 0, that alone is treated as success. Whether the workload actually had anything to process, or processed the expected volume of data, is not something Kubernetes checks.
The dev.to analysis points out that this lets a Job that "processed an empty queue" or "wrote a zero-byte backup" get marked as successful all the same. On top of that, automatic garbage collection driven by successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) means the logs and history you'd need to investigate disappear quickly too.
In other words, there's a fourth kind of silence here: "the Job did start, but effectively did nothing."
6. Turning This Into Monitoring That Actually Notices
What all four traps share is that Kubernetes' internal mechanics never self-report the anomaly. OneUptime's monitoring guide lays out the following approach:
- Regularly check the CronJob's
status.lastScheduleTime,status.lastSuccessfulTime, andstatus.active - Collect
kube_cronjob_status_last_schedule_timeandkube_cronjob_status_last_successful_time, exposed by kube-state-metrics, through Prometheus - Set alert thresholds at roughly twice the expected interval (around 48 hours for a daily job, around 2 hours for an hourly one)
- Pair this with external, cluster-independent monitoring (a dead man's switch) that can still detect an outage even when the cluster itself goes quiet
ZOZO TECH BLOG's case shows the same principle in practice: rather than relying solely on Kubernetes-native mechanisms, they built a threshold-based alerting setup on top of an external dashboard (same article). Given that the in-cluster controller can go silent, having a way to check "when did this last actually succeed?" from outside the cluster is the common defense against all four traps above.
Building this kind of monitoring stack from scratch — Prometheus and Grafana alone — takes real engineering effort. Kubo Cloud is a managed Kubernetes platform built on K3s that ships with Prometheus and Grafana included by default, so you don't have to build a CronJob status dashboard from zero.
Summary
CronJobs don't go silent because of a single misconfiguration — it's concurrencyPolicy, startingDeadlineSeconds, the 100-missed-schedule limit, and exit code 0 semantics combining into a silence that's hard to notice precisely because each piece behaves exactly as documented.
The real fix is to stop assuming "Kubernetes will tell me when something's wrong," and instead build in, from the start, external monitoring of status fields like lastScheduleTime and the metrics around them.
How much of that monitoring your team builds and maintains yourselves, versus how much you get from a platform that ships with it, is a question of where you want to spend engineering time. Kubo Cloud launches K3s clusters with Prometheus and Grafana already wired up, so the monitoring design covered in this article doesn't have to start from a blank slate. Take a look at the pricing plans and compare it against what your current CronJob operations actually cost you.
If you're running CronJobs on-premises or in a closed network where even setting up external monitoring is its own challenge, Kubo On-Premise is worth considering — it lets you build the same kind of monitoring design without sending data outside your own environment.