Claude Skill

k8s-prometheus-troubleshooting

Use when Prometheus on Kubernetes does not collect or alert as expected: a ServiceMonitor or PodMonitor whose targets never appear, targets that are down ("server returned HTTP status 404 Not Found", "connection refused", "context deadline exceeded"), a PrometheusRule that does n

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Download k8s-servant-k8s-skills-skills_k8s-prometheus-troubleshooting-0229910.zip · 9 KB

Install

skills CLI npx skills add https://github.com/k8s-servant/k8s-skills/tree/main/skills/k8s-prometheus-troubleshooting
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install k8s-servant-k8s-skills@llmmart
Git git clone https://github.com/k8s-servant/k8s-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole k8s-servant/k8s-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Prometheus troubleshooting

Find out why Prometheus does not scrape a target, does not load a rule, or does not alert, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

With the Prometheus Operator, a Prometheus custom resource selects ServiceMonitors, PodMonitors, Probes, ScrapeConfigs and PrometheusRules by label and namespace, and the operator turns them into Prometheus' configuration. A ServiceMonitor selects Services by label and names one of their ports. Something can go wrong at each step: the selection, the Service, the port, the scrape itself.

The commands use placeholders: monitoring (where Prometheus runs), my-prometheus (the Prometheus resource), my-namespace, my-app (the ServiceMonitor and its Service) and my-rules. Replace them with the real names. Prometheus' HTTP API is reached through the API server's service proxy on the governing Service prometheus-operated (port name web); this needs get on services/proxy, which the built-in view role does not have. If the proxy is forbidden, say so and ask the user for the Targets page of the Prometheus UI.

Never read Secrets. The operator stores the generated Prometheus and Alertmanager configuration in Secrets (prometheus-<name>, alertmanager-<name>-generated), and the upstream troubleshooting guide decodes them. Do not: use the selection rules, the operator's events and logs, and Prometheus' own API instead. For the same reason never call Prometheus' /api/v1/status/config or Alertmanager's /api/v2/status: they return that generated configuration (credentials masked, but webhook URLs and routes not). Access to the service proxy covers the whole Service, so this rule is yours to keep.

When to use

  • A new ServiceMonitor or PodMonitor shows no targets, or the job is missing.
  • Targets are down.
  • A PrometheusRule's alerts do not appear, do not fire, or do not reach Alertmanager.
  • The Prometheus or Alertmanager resource is not Available, or its pods crash.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The target's pods are not running or not Ready: k8s-workload-troubleshooting.
  • The target is up but a NetworkPolicy or DNS blocks the scrape: k8s-network-troubleshooting.
  • Prometheus' volume is full or will not bind: k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Read the Prometheus resource: its status, and which monitors and rules it selects.

kubectl get prometheuses -A
kubectl get prometheus my-prometheus -n monitoring -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}{.spec.serviceMonitorNamespaceSelector}{"\n"}{.spec.podMonitorSelector}{"\n"}{.spec.ruleSelector}{"\n"}{.spec.ruleNamespaceSelector}{"\n"}{.status.conditions}{"\n"}'

2. Read the ServiceMonitor and compare it with the selection and with the Service.

kubectl get servicemonitors,podmonitors -A
kubectl get servicemonitor my-app -n my-namespace -o yaml
kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.ports}{"\n"}'
kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app
kubectl get namespace my-namespace --show-labels

3. Ask Prometheus what it scrapes: the jobs it has, the active targets with their health and last error, and the targets it dropped.

kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/scrape_pools'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | [.labels.job, .health, .scrapeUrl, .lastError] | @tsv'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=dropped' | jq -r '.data.droppedTargets[].discoveredLabels | [.__meta_kubernetes_service_name, .__meta_kubernetes_endpoint_port_name, .__address__] | @tsv'

A ServiceMonitor appears as the job serviceMonitor/<namespace>/<name>/<endpoint index>.

4. Read the operator's events and logs. The operator rejects invalid objects with an event and logs objects it skips.

kubectl events -n my-namespace --for servicemonitor/my-app
kubectl get deployments -A -l app.kubernetes.io/name=prometheus-operator
kubectl logs -n monitoring deployment/prometheus-operator --tail=200 | grep -i -E "skipping|rejected|my-app"

5. Open the matching reference and follow its steps:

What you see Reference
No job or no targets for a ServiceMonitor or PodMonitor, dropped targets references/discovery.md
Targets down: HTTP status, connection refused, timeouts, body size, content type references/targets-down.md
A PrometheusRule not loaded, alerts not firing or not reaching Alertmanager references/rules-and-alerts.md
The Prometheus resource not Available, pods OOMKilled, disk full, ingestion errors references/prometheus-health.md

Read the results

What the common findings mean (Prometheus Operator 0.94, Prometheus 3.14):

  • No job serviceMonitor/<ns>/<name>/... in scrape_pools: the Prometheus does not select the ServiceMonitor. Its labels do not match serviceMonitorSelector, or its namespace does not match serviceMonitorNamespaceSelector. An empty selector ({}) matches everything; an unset (null) selector matches nothing for monitors and rules, and only Prometheus' own namespace for the namespace selectors.
  • The job exists, no active targets, a dropped target: the ServiceMonitor matches a Service, but its endpoints[].port is not the name of one of the Service's ports (it must be the port's name, not its number), or the Service has no ready endpoints. The dropped target's __meta_kubernetes_endpoint_port_name shows the real port name.
  • The job exists, no targets at all, nothing dropped: spec.selector matches no Service, or namespaceSelector leaves the Service's namespace out.
  • A target down with server returned HTTP status 404 Not Found: the path is wrong (endpoints[].path, default /metrics). Other lastError values are in references/targets-down.md.
  • An event InvalidConfiguration, "<name>" was rejected due to invalid configuration: ...: the operator refused the object; the message says why.

State the cause in one sentence, then the evidence you saw (the selector and labels, the port names, the target's health and error), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Patching a ServiceMonitor, PodMonitor or PrometheusRule (labels, port, path) is reversible, so it goes under "Ask first" and runs only after the user says yes. Changing the Prometheus or Alertmanager resource (selectors, resources, retention), Alertmanager's configuration and anything with Secrets are never run by you: hand them to the user as proposed fixes. Monitors usually come from the app's Helm chart or manifests; fix them there too. Write each undo with the values you found.

Give a ServiceMonitor the label the Prometheus selects:

Ask first: the agent runs this only after the user approves it.

kubectl label servicemonitor my-app -n my-namespace release=my-prometheus
# Undo: kubectl label servicemonitor my-app -n my-namespace release-

Point a ServiceMonitor at the Service's real port name:

Ask first: the agent runs this only after the user approves it.

kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "http"}]'
# Undo: kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "metrics"}]'

Prometheus picks up the change by itself within a minute or two; check scrape_pools and the targets again.

Sources

Checked against the Prometheus Operator 0.94 and Prometheus 3.14 documentation and source:

Files (k8s-skills)
  • .claude-plugin
    • plugin.json 1.1 KB
      Loading…
  • references
    • discovery.md 3.4 KB
      Loading…
    • prometheus-health.md 2.1 KB
      Loading…
    • rules-and-alerts.md 3.3 KB
      Loading…
    • targets-down.md 2.1 KB
      Loading…
  • SKILL.md 10.8 KB
    ---
    name: k8s-prometheus-troubleshooting
    description: >-
      Use when Prometheus on Kubernetes does not collect or alert as expected: a
      ServiceMonitor or PodMonitor whose targets never appear, targets that are down
      ("server returned HTTP status 404 Not Found", "connection refused", "context deadline
      exceeded"), a PrometheusRule that does not load or alerts that do not fire or reach
      Alertmanager, a Prometheus or Alertmanager custom resource that is not Available,
      "InvalidConfiguration" events from the Prometheus Operator, or Prometheus pods that are
      OOMKilled or out of disk. Covers the Prometheus Operator (kube-prometheus,
      kube-prometheus-stack). Investigates with kubectl get, describe, events, logs and
      Prometheus' HTTP API through the API server, and proposes fixes. Reads first and asks
      before changing anything.
    license: Apache-2.0
    metadata:
      verified-with: "kubernetes 1.37, prometheus-operator 0.94, prometheus 3.14"
      verified-on: "2026-10-24"
    ---
    
    # Prometheus troubleshooting
    
    Find out why Prometheus does not scrape a target, does not load a rule, or does not
    alert, and propose a fix. **Read first, ask before changing.** Run the investigation
    commands freely. Run a command under an "Ask first" line only after the user approves
    that exact command; if your credentials do not allow it, give it to the user to run.
    Never run a "Proposed fix"; hand it to the user.
    
    With the **Prometheus Operator**, a `Prometheus` custom resource selects
    **ServiceMonitors**, **PodMonitors**, **Probes**, **ScrapeConfigs** and
    **PrometheusRules** by label and namespace, and the operator turns them into
    Prometheus' configuration. A ServiceMonitor selects Services by label and names one of
    their ports. Something can go wrong at each step: the selection, the Service, the port,
    the scrape itself.
    
    The commands use placeholders: `monitoring` (where Prometheus runs), `my-prometheus`
    (the Prometheus resource), `my-namespace`, `my-app` (the ServiceMonitor and its
    Service) and `my-rules`. Replace them with the real names. Prometheus' HTTP API is
    reached through the API server's service proxy on the governing Service
    `prometheus-operated` (port name `web`); this needs `get` on `services/proxy`, which the
    built-in `view` role does not have. If the proxy is forbidden, say so and ask the user
    for the Targets page of the Prometheus UI.
    
    Never read Secrets. The operator stores the generated Prometheus and Alertmanager
    configuration in Secrets (`prometheus-<name>`, `alertmanager-<name>-generated`), and
    the upstream troubleshooting guide decodes them. Do not: use the selection rules, the
    operator's events and logs, and Prometheus' own API instead. For the same reason never
    call Prometheus' `/api/v1/status/config` or Alertmanager's `/api/v2/status`: they return
    that generated configuration (credentials masked, but webhook URLs and routes not).
    Access to the service proxy covers the whole Service, so this rule is yours to keep.
    
    ## When to use
    
    - A new ServiceMonitor or PodMonitor shows no targets, or the job is missing.
    - Targets are `down`.
    - A PrometheusRule's alerts do not appear, do not fire, or do not reach Alertmanager.
    - The `Prometheus` or `Alertmanager` resource is not `Available`, or its pods crash.
    
    Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill;
    if it is not installed, keep going with the steps here and tell the user which skill
    would help:
    
    - The target's pods are not running or not Ready: `k8s-workload-troubleshooting`.
    - The target is up but a NetworkPolicy or DNS blocks the scrape:
      `k8s-network-troubleshooting`.
    - Prometheus' volume is full or will not bind: `k8s-storage-troubleshooting`.
    
    ## Investigate
    
    Work through the steps in order. Stop when you have the cause and its evidence.
    
    **1. Read the Prometheus resource**: its status, and which monitors and rules it
    selects.
    
    ```bash
    kubectl get prometheuses -A
    kubectl get prometheus my-prometheus -n monitoring -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}{.spec.serviceMonitorNamespaceSelector}{"\n"}{.spec.podMonitorSelector}{"\n"}{.spec.ruleSelector}{"\n"}{.spec.ruleNamespaceSelector}{"\n"}{.status.conditions}{"\n"}'
    ```
    
    **2. Read the ServiceMonitor** and compare it with the selection and with the Service.
    
    ```bash
    kubectl get servicemonitors,podmonitors -A
    kubectl get servicemonitor my-app -n my-namespace -o yaml
    kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.ports}{"\n"}'
    kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app
    kubectl get namespace my-namespace --show-labels
    ```
    
    **3. Ask Prometheus what it scrapes**: the jobs it has, the active targets with their
    health and last error, and the targets it dropped.
    
    ```bash
    kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/scrape_pools'
    kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | [.labels.job, .health, .scrapeUrl, .lastError] | @tsv'
    kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=dropped' | jq -r '.data.droppedTargets[].discoveredLabels | [.__meta_kubernetes_service_name, .__meta_kubernetes_endpoint_port_name, .__address__] | @tsv'
    ```
    
    A ServiceMonitor appears as the job `serviceMonitor/<namespace>/<name>/<endpoint index>`.
    
    **4. Read the operator's events and logs.** The operator rejects invalid objects with an
    event and logs objects it skips.
    
    ```bash
    kubectl events -n my-namespace --for servicemonitor/my-app
    kubectl get deployments -A -l app.kubernetes.io/name=prometheus-operator
    kubectl logs -n monitoring deployment/prometheus-operator --tail=200 | grep -i -E "skipping|rejected|my-app"
    ```
    
    **5. Open the matching reference** and follow its steps:
    
    | What you see | Reference |
    |---|---|
    | No job or no targets for a ServiceMonitor or PodMonitor, dropped targets | [references/discovery.md](references/discovery.md) |
    | Targets `down`: HTTP status, `connection refused`, timeouts, body size, content type | [references/targets-down.md](references/targets-down.md) |
    | A PrometheusRule not loaded, alerts not firing or not reaching Alertmanager | [references/rules-and-alerts.md](references/rules-and-alerts.md) |
    | The Prometheus resource not Available, pods OOMKilled, disk full, ingestion errors | [references/prometheus-health.md](references/prometheus-health.md) |
    
    ## Read the results
    
    What the common findings mean (Prometheus Operator 0.94, Prometheus 3.14):
    
    - **No job `serviceMonitor/<ns>/<name>/...` in `scrape_pools`**: the Prometheus does not
      select the ServiceMonitor. Its labels do not match `serviceMonitorSelector`, or its
      namespace does not match `serviceMonitorNamespaceSelector`. An empty selector (`{}`)
      matches everything; an unset (null) selector matches nothing for monitors and rules,
      and only Prometheus' own namespace for the namespace selectors.
    - **The job exists, no active targets, a dropped target**: the ServiceMonitor matches a
      Service, but its `endpoints[].port` is not the name of one of the Service's ports (it
      must be the port's `name`, not its number), or the Service has no ready endpoints.
      The dropped target's `__meta_kubernetes_endpoint_port_name` shows the real port name.
    - **The job exists, no targets at all, nothing dropped**: `spec.selector` matches no
      Service, or `namespaceSelector` leaves the Service's namespace out.
    - **A target `down` with `server returned HTTP status 404 Not Found`**: the path is
      wrong (`endpoints[].path`, default `/metrics`). Other `lastError` values are in
      [references/targets-down.md](references/targets-down.md).
    - **An event `InvalidConfiguration`, `"<name>" was rejected due to invalid
      configuration: ...`**: the operator refused the object; the message says why.
    
    State the cause in one sentence, then the evidence you saw (the selector and labels, the
    port names, the target's health and error), then the proposed fix.
    
    ## Proposed fix
    
    Show each fix to the user before running it: what it changes and how to undo it.
    Patching a ServiceMonitor, PodMonitor or PrometheusRule (labels, port, path) is
    reversible, so it goes under "Ask first" and runs only after the user says yes.
    Changing the Prometheus or Alertmanager resource (selectors, resources, retention),
    Alertmanager's configuration and anything with Secrets are never run by you: hand them
    to the user as proposed fixes. Monitors usually come from the app's Helm chart or
    manifests; fix them there too. Write each undo with the values you found.
    
    Give a ServiceMonitor the label the Prometheus selects:
    
    > Ask first: the agent runs this only after the user approves it.
    
    ```bash
    kubectl label servicemonitor my-app -n my-namespace release=my-prometheus
    # Undo: kubectl label servicemonitor my-app -n my-namespace release-
    ```
    
    Point a ServiceMonitor at the Service's real port name:
    
    > Ask first: the agent runs this only after the user approves it.
    
    ```bash
    kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "http"}]'
    # Undo: kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "metrics"}]'
    ```
    
    Prometheus picks up the change by itself within a minute or two; check `scrape_pools`
    and the targets again.
    
    ## Sources
    
    Checked against the Prometheus Operator 0.94 and Prometheus 3.14 documentation and
    source:
    
    - [Prometheus Operator troubleshooting](https://prometheus-operator.dev/docs/platform/troubleshooting/):
      selection, port names, events; its step that decodes the configuration Secret is
      left out here.
    - [API reference](https://prometheus-operator.dev/docs/api-reference/api/): selector
      semantics (empty versus null), ServiceMonitor endpoints.
    - [RBAC for the CRDs](https://prometheus-operator.dev/docs/platform/rbac-crd/) and
      [storage](https://prometheus-operator.dev/docs/platform/storage/).
    - Prometheus [HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/)
      (targets, scrape pools, rules, alerts, TSDB status),
      [storage](https://prometheus.io/docs/prometheus/latest/storage/) and the
      [3.0 migration guide](https://prometheus.io/docs/prometheus/latest/migration/).
    - [Manually constructing API server proxy URLs](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster-services/#manually-constructing-apiserver-proxy-urls).
    - Messages from the source:
      [resource_selector.go](https://github.com/prometheus-operator/prometheus-operator/blob/v0.94.2/pkg/prometheus/resource_selector.go),
      [operator.go](https://github.com/prometheus-operator/prometheus-operator/blob/v0.94.2/pkg/operator/operator.go),
      [scrape.go](https://github.com/prometheus/prometheus/blob/v3.14.0/scrape/scrape.go).
      The behaviour and messages quoted here were also seen on a Kubernetes 1.37 kind
      cluster with Prometheus Operator 0.94.2 and Prometheus 3.14.0.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.