We Killed One K3s Node, But Grafana Still Said "Running"
Deploying K3s clusters across multiple sites — factories, retail stores — and shipping metrics to a central Prometheus/Grafana stack is becoming a standard edge pattern. That tracks with K3s being designed as a production-ready Kubernetes distribution for unattended, resource-constrained, remote locations.
But when you run Prometheus in Agent mode on the site side, something strange can happen after you deliberately take down a node or the network drops: the central dashboard keeps showing the "last known value" as if everything were still running. That's how an incident starts with "the dashboard is green, but the site just told us it's actually down."
This article walks through a trap that shows up in practice when using K3s Prometheus Agent mode as the edge-side monitoring agent — metrics that appear to survive a target's disappearance — and how to design around it.
Agent Mode Isn't Just "Scrape and Forward"
If you run a full Prometheus server on a resource-constrained environment like K3s, you pay the memory and disk cost of the local TSDB (time-series database), the query engine, and rule evaluation. Prometheus's Agent mode exists to avoid exactly that.
Per the official Prometheus documentation, Agent mode is a dedicated operating mode focused on scraping and remote writing, with normal TSDB queries, recording rules, and alerting disabled. It uses a customized WAL (Write-Ahead Log) that deletes data immediately after a successful remote write, so it doesn't need to build in-memory chunks or maintain query indexes — which is why it uses substantially fewer resources than a standard Prometheus server.
The official Prometheus blog post introducing the feature explains that the rise of resource-constrained environments — edge clusters, ephemeral Kubernetes clusters — and the growing need to centralize monitoring data drove the development of Agent mode. K3s itself is built on the same resource-conscious philosophy: it's a single binary with a small memory footprint that can run on low-spec hardware like a Raspberry Pi, and the monitoring agent at each site needs that same discipline.
There's a catch, though. The official docs state that the disk buffer for when a remote endpoint is unreachable is currently limited to a two-hour window. For sites with long network outages, that limit translates directly into data loss.
The "Stale Marker" Mechanism Behind a Disappearing Target
Here's the core issue. K3s edge nodes disappear far more often than standard clusters running in the cloud: node restarts, spot-style operation, brief network blips. Every time one of these happens, a Prometheus scrape target vanishes — temporarily or for good.
The Prometheus remote write protocol has a mechanism for exactly this situation. The official Prometheus Remote Write specification states that compliant senders "MUST send stale markers when a time series will no longer be appended to." This is signaled using a special NaN value (0x7ff0000000000002) that must not be used for any other purpose. The spec lists several triggers for sending this marker, including when service discovery detects that a target has disappeared, or when a time series is no longer exposed between consecutive scrapes.
In other words: if a target's disappearance correctly triggers a stale marker sent to the central remote storage, the system knows "this series will no longer be updated," and the dashboard correctly shows it as having no data. But if the Agent process itself stops responding during a network blip, or the two-hour disk buffer limit is exceeded, the stale marker never arrives — and the remote side is left holding the "last good value." Visually, it's a graph that looks green and alive, while nothing is actually being measured anymore. That's the real cause of the scenario at the top of this article.
K3s's own edge clusters expose a wide range of metrics — cluster management, work queues, node components — through a single /metrics endpoint, which means reachability of that endpoint is itself an important liveness signal for a site. If you introduce Agent mode without understanding stale marker behavior, you risk breaking the very mechanism meant to tell you a site has gone down.
A Design Checklist for Running Agent Mode Across Multiple K3s Edge Sites
If you're rolling out Prometheus Agent mode across multiple K3s edge sites in practice, there are at least three things worth confirming at the design stage.
1. Can you identify which site a cluster belongs to, via labels?
When multiple sites write to the same remote storage, you need a way to tell which metrics came from which site — otherwise you can't isolate incidents. The official Grafana documentation describes attaching a cluster-level label to metrics via external_labels, as well as routing by tenant using an HTTP header like X-Scope-OrgID. For an edge setup that assumes a growing number of sites, you should always attach a site-ID-equivalent label through external_labels.
# Example K3s edge-side Prometheus Agent mode config (excerpt)
global:
external_labels:
site_id: "factory-osaka-03"
cluster: "k3s-edge"
remote_write:
- url: "https://central-prometheus.example.com/api/v1/write"
2. Are stale markers disabled anywhere?
Some implementations let you disable stale marker transmission to save resources, but you should not do that in an edge environment where sites come and go frequently. "Data stopped arriving" is itself a critical failure signal, and suppressing it turns the "looks alive" problem described above into a recurring issue.
3. Is your buffer sized for the longest network outage you expect?
As noted above, the Agent mode disk buffer is capped at two hours. For sites with long overnight maintenance windows where the network goes down, you should design around the assumption that data beyond two hours will be lost — for example, by building in an alert suppression window right after recovery instead of relying on retransmission.
As the number of sites grows, checking these three items one by one by hand becomes impractical. A management view like Kubo Cloud's Captain UI, which visualizes configuration and labeling status across clusters at a glance, makes it easier to spot which site is missing a label or has inconsistent stale marker settings.
Conclusion: Treat Agent Mode as a Different Component, Not a "Lightweight Swap"
Prometheus Agent mode is a solid option for keeping monitoring costs down on resource-constrained edge clusters like K3s. It's reassuring that this lightweight forwarding mode exists as part of an ecosystem that graduated within the CNCF in 2018, right after Kubernetes.
But "lightweight" doesn't mean "simple." Stale marker behavior on target disappearance, the two-hour disk buffer limit, label design for identifying multiple sites — skip understanding any of these, and you end up in a state where you think you're monitoring, but you're actually not seeing anything.
The operational cost of working out every one of these details yourself only grows as you add more sites. Kubo, a K3s-based managed Kubernetes offering, ships with a Prometheus + Grafana monitoring stack out of the box, built around these exact pitfalls. Rather than wiring up Agent mode, stale markers, and label design one piece at a time, you can start from a configuration that already accounts for them.
For manufacturing DX use cases with many edge sites, Kubo On-Premise lets you run your site-side K3s clusters and monitoring stack together without having to move data outside your own environment. If you'd like to talk through your site rollout plans, you can reach out via Contact Us.