Prometheus Is Running. The Metrics Aren't. Three Reasons Your ServiceMonitor Fails Silently

Prometheus Is Up. The Dashboard Loads Fine. The Graph Is Just Empty

You installed kube-prometheus-stack with Helm, and logging into Grafana worked without a hitch. But when you try to view your application's custom metrics, the panel stays blank. No error logs. No alerts firing. The data simply isn't there.

This is a classic "silent failure" in Kubernetes monitoring setups. A kubernetes service monitor offers more flexible configuration than Pod annotation-based discovery, but as the Prometheus Operator official documentation makes clear, it's designed to work only when multiple layers of conditions align. If even one layer is off, metrics disappear without a sound.

There's a structural irony at the heart of monitoring: your monitoring stack catches application failures, but nothing catches monitoring failures themselves. In this article, we'll walk through the three most common reasons a ServiceMonitor silently drops metrics, along with a step-by-step way to diagnose each one. As a side note, if you'd rather not own this initial setup burden at all, managed K3s environments like Kubo handle it for you out of the box — but it's worth understanding the mechanics first.

Reason 1: The "Triple Label Match" Between Service, ServiceMonitor, and the Prometheus CR Is Broken

According to the official documentation, a ServiceMonitor's selector field is meant to "select Kubernetes Pod objects to scrape metrics from, using label selectors." In other words, it's not enough to put labels on the ServiceMonitor itself — those labels must exactly match the labels on the Service being monitored.

It gets more complex if you're using the kube-prometheus-stack Helm chart. By default, the Prometheus instance only discovers ServiceMonitors that are "in the same namespace as itself AND carry the same release label as prometheus-operator." That means:

1. The ServiceMonitor's selector.matchLabels must match the Service's labels
2. The Prometheus CR's serviceMonitorSelector must match the ServiceMonitor's labels (the release label)

If either of these two label matches fails, the ServiceMonitor exists but is effectively invisible to Prometheus. On the Helm chart side, setting serviceMonitorSelectorNilUsesHelmValues to false disables the release-specific label filtering and makes Prometheus target every ServiceMonitor in the namespace — but that comes with the tradeoff of unintentionally picking up unrelated ServiceMonitors.

What to Check

• Does the ServiceMonitor's spec.selector.matchLabels exactly match the Service's metadata.labels?
• In a kube-prometheus-stack environment, does the ServiceMonitor itself carry the correct release: {release-name} label?
• What labels does the Prometheus CR's spec.serviceMonitorSelector require? Check with kubectl get prometheus -o yaml.

Reason 2: Without a namespaceSelector, Prometheus Only Sees Its Own Namespace

It's common to separate your application into a production namespace and your monitoring stack into a monitoring namespace. But in this setup, forgetting to configure namespaceSelector means your metrics will never be collected — indefinitely.

The official spec is explicit here. According to the ServiceMonitor API reference, an empty label selector for namespaceSelector will detect Pods across all namespaces, but a null selector (i.e., left unset) means only Pods in the same namespace as the ServiceMonitor object are targeted. This is the default behavior — cross-namespace monitoring simply doesn't happen without explicit configuration.

If you don't know this, you'll run straight into a scenario where "I created the ServiceMonitor in the monitoring namespace, but it's not picking up the Service in the production namespace." Even if all the label matching from Reason 1 is perfect, none of it matters if you haven't crossed this namespace boundary.

The Fix

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s

Use matchNames to explicitly list target namespaces, or set any: true if you need to monitor across all namespaces.

Reason 3: The Labels Are Right, but RBAC Permissions Make It "Visible Yet Invisible"

The first two causes are usually caught by reviewing YAML, but the third is the one that's most often missed. Labels are perfect, namespaceSelector is perfect, and metrics still don't show up.

The cause lies in insufficient RBAC permissions on Prometheus's ServiceAccount. As the Kubernetes ServiceAccount documentation states, the auto-created default ServiceAccount has almost no permissions beyond "default API discovery access." Unless Prometheus is explicitly granted permission to list/watch Services, Endpoints, and Pods in the target namespace, it won't even be recognized as a scrape target.

According to the Kubernetes RBAC documentation, permissions are granted by binding a Role's defined verbs (get, list, watch, etc.) to a ServiceAccount through a RoleBinding. It's also important to note that RoleBindings are namespace-scoped while ClusterRoleBindings are cluster-wide. For multi-namespace monitoring, you'll need to decide whether to add a RoleBinding per namespace or grant access in bulk with a ClusterRole and ClusterRoleBinding.

What makes this issue particularly tricky is that it never surfaces as an error. Prometheus doesn't crash when it lacks permission to list a resource — the target simply never appears in the target list, and you won't notice unless you're carefully reading logs.

Verification Commands

kubectl auth can-i list services --as=system:serviceaccount:monitoring:prometheus-kube-prometheus-prometheus -n production
kubectl auth can-i list endpoints --as=system:serviceaccount:monitoring:prometheus-kube-prometheus-prometheus -n production

If either returns no, you need to add a Role/RoleBinding for the target namespace. As the Kubernetes RBAC good practices guide also recommends, the right approach is to follow the principle of least privilege while explicitly granting only the access monitoring actually needs.

A Practical Diagnostic Workflow — From the Targets Page to kubectl Commands in 4 Steps

Knowing there are three possible causes doesn't tell you where to start when something's actually broken in production. Follow this order and you won't waste time guessing.

1. Open the Prometheus UI's Targets page (/targets). Check whether your job appears in the list at all. If it's missing, suspect Reason 1 or 2; if it appears but shows DOWN, suspect the endpoint itself.
2. Check the ServiceMonitor's selector. Run kubectl get servicemonitor <name> -o yaml to print spec.selector and spec.namespaceSelector, and cross-check them against the target Service's labels.
3. Check the Prometheus CR's selector. Run kubectl get prometheus -n monitoring -o yaml to check spec.serviceMonitorSelector and confirm it matches the ServiceMonitor's own labels.
4. Check RBAC permissions with kubectl auth can-i. If everything above matches but the target still isn't showing up, RBAC is almost certainly the cause.

Following these steps lets you mechanically pinpoint exactly which of the three causes is at fault. Put another way: unless you turn these four steps into a standard procedure up front, you'll end up re-investigating the same root causes from scratch every time.

Prometheus was the second project ever to graduate from the CNCF, and precisely because it's become the de facto standard in the Kubernetes ecosystem, this configuration complexity is something most teams run into at least once. The CNCF's overview of Kubernetes monitoring also points to the importance of configuration consistency when combining multiple open-source toolsets.

If doing this triage yourself every time feels like a burden, Kubo ships with Prometheus + Grafana pre-installed, in a configuration where ServiceMonitor label alignment and RBAC setup have already been verified to work. It's built on a lightweight K3s foundation and takes on the entire process of building a monitoring stack from scratch.

Designing With the Difference Between Monitoring and Observability in Mind

Considering the difference between monitoring and observability, monitoring is specifically about detecting "clearly defined failures," while observability refers to the broader goal of achieving visibility through logs, metrics, and traces combined. A ServiceMonitor misconfiguration means the very foundation that monitoring is supposed to provide isn't functioning — which means the entire observability layer built on top of it collapses too.

The AWS documentation on operational excellence also positions implementing observability as one of the core pillars of operations. A gap in your monitoring foundation isn't just a technical mistake — it's an issue that affects the reliability of your entire operation.

Summary — Work Through These Three Causes in Order, and You Can Prevent Monitoring's "Silent Failure"

When Prometheus + ServiceMonitor isn't collecting metrics, the cause almost always comes down to one of these three:

• Label selector mismatch: Has the triple alignment between Service, ServiceMonitor, and the Prometheus CR broken down somewhere?
• Missing namespaceSelector configuration: Do you understand that, by default, Prometheus can only see its own namespace?
• Insufficient RBAC permissions: Does the ServiceAccount explicitly have list/watch access?

All three are documented as explicit specifications in the official docs, but because the configuration spans multiple CRDs and RBAC, they're easy to overlook when something actually breaks in production. If you'd rather stop manually checking these three things every time, it's worth considering a managed K3s environment like Kubo, which comes with Prometheus + Grafana pre-installed and pre-configured out of the box. Compared to building a monitoring stack from scratch on EKS or AKS, it lets you run production-grade Kubernetes more cost-effectively.

If you're wrestling with these kinds of pitfalls in building or operating your monitoring infrastructure, feel free to get in touch — we're happy to talk it through.