Claude Skill

k8s-grafana-troubleshooting

Use when Grafana on Kubernetes fails: the Grafana pod crash-loops after a change to its provisioning ("Datasource provisioning error", "Only one datasource per organization can be marked as default", "failed to parse"), a provisioned dashboard or datasource does not appear ("fail

LLM Mart · 0 points · 1 views 0 listing impressions 0 install-command copies
Download k8s-servant-k8s-skills-skills_k8s-grafana-troubleshooting-0229910.zip · 9 KB

Install

skills CLI npx skills add https://github.com/k8s-servant/k8s-skills/tree/main/skills/k8s-grafana-troubleshooting
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install k8s-servant-k8s-skills@llmmart
Git git clone https://github.com/k8s-servant/k8s-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole k8s-servant/k8s-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Grafana troubleshooting

Find out why Grafana does not start, does not show a datasource or dashboard, or shows no data, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

On Kubernetes, Grafana is usually provisioned: datasources and dashboard providers are YAML files under /etc/grafana/provisioning/, and dashboards are JSON files, all mounted from ConfigMaps. With the Grafana Helm chart (also inside kube-prometheus-stack) a sidecar container copies ConfigMaps labelled grafana_dashboard or grafana_datasource into the pod. With the grafana-operator, Grafana, GrafanaDashboard and GrafanaDatasource resources do the same.

The commands use placeholders: monitoring (where Grafana runs), my-grafana (its Deployment and Service), my-grafana-datasources and my-grafana-dashboards (its ConfigMaps) and my-app (a dashboard). Replace them with the real names.

Never read Secrets: not the admin password, not datasource credentials, not grafana.ini values kept in Secrets. The sidecar can also read Secrets with the same labels; look only at ConfigMaps. Do not log in to Grafana's API with the admin password; its /api/health needs no login.

When to use

  • The Grafana pod is in CrashLoopBackOff or not Ready, often right after a change.
  • A provisioned datasource or dashboard is missing in the UI.
  • Panels show "No data", or a datasource test fails.
  • Logins fail, or the sidecar logs 401 Unauthorized.
  • grafana-operator resources do not sync.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • Grafana runs and reaches Prometheus, but Prometheus has no data for the query: k8s-prometheus-troubleshooting.
  • The datasource's Service exists but traffic does not reach it (NetworkPolicy, DNS): k8s-network-troubleshooting.
  • The pod is Pending or its volume does not mount: k8s-workload-troubleshooting, k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the pods. During a rollout the old pod can keep running while the new one crashes; look at each pod, not only the Deployment.

kubectl get deployments,pods -n monitoring -l app=my-grafana -o wide
kubectl get pods -n monitoring -l app=my-grafana -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{"\n"}{range .status.containerStatuses[*]}{"  "}{.name}{" ready="}{.ready}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}{end}'
kubectl events -n monitoring

Avoid kubectl describe pod here: it prints the containers' environment values, and Grafana pods often carry passwords there. If you see a credential anyway, do not repeat it.

The chart labels its pods app.kubernetes.io/name=grafana; use that selector there.

2. Read the logs of the failing pod, the previous run if it restarted. Each pod has the grafana container, and with the chart one container per sidecar (grafana-sc-dashboard, grafana-sc-datasources).

kubectl logs -n monitoring my-grafana-0 -c grafana --previous --tail=50
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=200 | grep -E "level=error|level=warn|Error:"
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=500 | grep logger=provisioning

Replace my-grafana-0 with the failing pod's name from step 1.

3. Read the provisioning ConfigMaps the pod mounts.

kubectl get deployment my-grafana -n monitoring -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{"\n"}{end}'
kubectl get configmap my-grafana-datasources -n monitoring -o yaml
kubectl get configmaps -n monitoring -l grafana_dashboard
kubectl get configmaps -n monitoring -l grafana_datasource

4. For "No data" or a failing datasource, check the address it uses against the Services that exist.

kubectl get services -A | grep -i -E "prometheus|loki|tempo"
kubectl get endpointslices -n monitoring
kubectl get --raw '/api/v1/namespaces/monitoring/services/my-grafana:http/proxy/api/health'

The last command needs get on services/proxy; if it is forbidden, skip it.

5. Open the matching reference and follow its steps:

What you see Reference
CrashLoopBackOff, Datasource provisioning error, failed to parse, a datasource or dashboard missing, sidecar not picking up ConfigMaps references/provisioning.md
"No data", Bad Gateway, connection refused, wrong datasource UID in a dashboard references/datasources-no-data.md
database is locked, permission errors, plugins, login and 401 Unauthorized, grafana-operator NoMatchingInstance references/runtime-and-operator.md

Read the results

What the common findings mean (Grafana 13.2):

  • The new pod exits with Error: ✗ invalid service state: Failed ... Datasource provisioning error: <reason>: Grafana refuses a provisioned datasource file and stops; the pod crash-loops. datasource.yaml config is invalid. Only one datasource per organization can be marked as default means two datasources have isDefault: true. A YAML error or an unknown field shows up the same way.
  • logger=provisioning.dashboard ... level=error msg="failed to load dashboard from " file=<path> error=<JSON error>: that dashboard file is not valid JSON. Grafana skips it and keeps running, so only that dashboard is missing.
  • the same UID is used more than once or dashboard title is not unique in folder: two files define the same dashboard; only one wins.
  • A datasource whose URL names a Service that does not exist (or the wrong port): Grafana starts normally; every query fails, shown in the UI as "Bad Gateway" or a connection error, and in the log only when a query runs.
  • /api/health returns "database": "ok": Grafana itself is up; failing (HTTP 503) means its database is not reachable.

State the cause in one sentence, then the evidence you saw (the log line, the ConfigMap entry, the Services), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Provisioning lives in ConfigMaps that come from a Helm chart, kube-prometheus-stack values or a GitOps repository: fix it there. Patching a ConfigMap that only holds provisioning or dashboards, and restarting Grafana to reload it, are reversible, so they go under "Ask first" and run only after the user says yes. Anything with Secrets (admin password, datasource credentials), Grafana's database, its volume and the replica count are never run by you: hand them to the user as proposed fixes. Write each undo with the values you found.

Restart Grafana after its provisioning ConfigMap is fixed (Grafana reads datasource files only at start):

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-grafana -n monitoring
# Undo: kubectl rollout undo deployment/my-grafana -n monitoring

Sources

Checked against the Grafana 13.2 documentation and source (v13.2.4), the Grafana Helm chart 14.0 and grafana-operator 5.25:

Files (k8s-skills)
  • .claude-plugin
    • plugin.json 1.1 KB
      Loading…
  • references
    • datasources-no-data.md 2.1 KB
      Loading…
    • provisioning.md 3.4 KB
      Loading…
    • runtime-and-operator.md 3.3 KB
      Loading…
  • SKILL.md 9.8 KB
    ---
    name: k8s-grafana-troubleshooting
    description: >-
      Use when Grafana on Kubernetes fails: the Grafana pod crash-loops after a change to its
      provisioning ("Datasource provisioning error", "Only one datasource per organization
      can be marked as default", "failed to parse"), a provisioned dashboard or datasource
      does not appear ("failed to load dashboard from", sidecar ConfigMap labels), panels
      show "No data" or the datasource returns "Bad Gateway" or "connection refused",
      "database is locked" with several replicas, permission errors on the data volume,
      admin login or sidecar "401 Unauthorized", or grafana-operator resources with
      "NoMatchingInstance". Investigates with kubectl get, describe, events and logs, never
      reading Grafana's Secrets, and proposes fixes. Reads first and asks before changing
      anything.
    license: Apache-2.0
    metadata:
      verified-with: "kubernetes 1.37, grafana 13.2"
      verified-on: "2026-10-24"
    ---
    
    # Grafana troubleshooting
    
    Find out why Grafana does not start, does not show a datasource or dashboard, or shows
    no data, and propose a fix. **Read first, ask before changing.** Run the investigation
    commands freely. Run a command under an "Ask first" line only after the user approves
    that exact command; if your credentials do not allow it, give it to the user to run.
    Never run a "Proposed fix"; hand it to the user.
    
    On Kubernetes, Grafana is usually **provisioned**: datasources and dashboard providers
    are YAML files under `/etc/grafana/provisioning/`, and dashboards are JSON files, all
    mounted from ConfigMaps. With the Grafana Helm chart (also inside kube-prometheus-stack)
    a **sidecar** container copies ConfigMaps labelled `grafana_dashboard` or
    `grafana_datasource` into the pod. With the **grafana-operator**, `Grafana`,
    `GrafanaDashboard` and `GrafanaDatasource` resources do the same.
    
    The commands use placeholders: `monitoring` (where Grafana runs), `my-grafana` (its
    Deployment and Service), `my-grafana-datasources` and `my-grafana-dashboards` (its
    ConfigMaps) and `my-app` (a dashboard). Replace them with the real names.
    
    Never read Secrets: not the admin password, not datasource credentials, not
    `grafana.ini` values kept in Secrets. The sidecar can also read Secrets with the same
    labels; look only at ConfigMaps. Do not log in to Grafana's API with the admin
    password; its `/api/health` needs no login.
    
    ## When to use
    
    - The Grafana pod is in CrashLoopBackOff or not Ready, often right after a change.
    - A provisioned datasource or dashboard is missing in the UI.
    - Panels show "No data", or a datasource test fails.
    - Logins fail, or the sidecar logs `401 Unauthorized`.
    - grafana-operator resources do not sync.
    
    Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill;
    if it is not installed, keep going with the steps here and tell the user which skill
    would help:
    
    - Grafana runs and reaches Prometheus, but Prometheus has no data for the query:
      `k8s-prometheus-troubleshooting`.
    - The datasource's Service exists but traffic does not reach it (NetworkPolicy, DNS):
      `k8s-network-troubleshooting`.
    - The pod is Pending or its volume does not mount: `k8s-workload-troubleshooting`,
      `k8s-storage-troubleshooting`.
    
    ## Investigate
    
    Work through the steps in order. Stop when you have the cause and its evidence.
    
    **1. See the pods.** During a rollout the old pod can keep running while the new one
    crashes; look at each pod, not only the Deployment.
    
    ```bash
    kubectl get deployments,pods -n monitoring -l app=my-grafana -o wide
    kubectl get pods -n monitoring -l app=my-grafana -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{"\n"}{range .status.containerStatuses[*]}{"  "}{.name}{" ready="}{.ready}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}{end}'
    kubectl events -n monitoring
    ```
    
    Avoid `kubectl describe pod` here: it prints the containers' environment values, and
    Grafana pods often carry passwords there. If you see a credential anyway, do not repeat
    it.
    
    The chart labels its pods `app.kubernetes.io/name=grafana`; use that selector there.
    
    **2. Read the logs of the failing pod**, the previous run if it restarted. Each pod has
    the `grafana` container, and with the chart one container per sidecar
    (`grafana-sc-dashboard`, `grafana-sc-datasources`).
    
    ```bash
    kubectl logs -n monitoring my-grafana-0 -c grafana --previous --tail=50
    kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=200 | grep -E "level=error|level=warn|Error:"
    kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=500 | grep logger=provisioning
    ```
    
    Replace `my-grafana-0` with the failing pod's name from step 1.
    
    **3. Read the provisioning ConfigMaps** the pod mounts.
    
    ```bash
    kubectl get deployment my-grafana -n monitoring -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{"\n"}{end}'
    kubectl get configmap my-grafana-datasources -n monitoring -o yaml
    kubectl get configmaps -n monitoring -l grafana_dashboard
    kubectl get configmaps -n monitoring -l grafana_datasource
    ```
    
    **4. For "No data" or a failing datasource, check the address it uses** against the
    Services that exist.
    
    ```bash
    kubectl get services -A | grep -i -E "prometheus|loki|tempo"
    kubectl get endpointslices -n monitoring
    kubectl get --raw '/api/v1/namespaces/monitoring/services/my-grafana:http/proxy/api/health'
    ```
    
    The last command needs `get` on `services/proxy`; if it is forbidden, skip it.
    
    **5. Open the matching reference** and follow its steps:
    
    | What you see | Reference |
    |---|---|
    | CrashLoopBackOff, `Datasource provisioning error`, `failed to parse`, a datasource or dashboard missing, sidecar not picking up ConfigMaps | [references/provisioning.md](references/provisioning.md) |
    | "No data", Bad Gateway, `connection refused`, wrong datasource UID in a dashboard | [references/datasources-no-data.md](references/datasources-no-data.md) |
    | `database is locked`, permission errors, plugins, login and `401 Unauthorized`, grafana-operator `NoMatchingInstance` | [references/runtime-and-operator.md](references/runtime-and-operator.md) |
    
    ## Read the results
    
    What the common findings mean (Grafana 13.2):
    
    - **The new pod exits with `Error: ✗ invalid service state: Failed ... Datasource
      provisioning error: <reason>`**: Grafana refuses a provisioned datasource file and
      stops; the pod crash-loops. `datasource.yaml config is invalid. Only one datasource
      per organization can be marked as default` means two datasources have `isDefault:
      true`. A YAML error or an unknown field shows up the same way.
    - **`logger=provisioning.dashboard ... level=error msg="failed to load dashboard from "
      file=<path> error=<JSON error>`**: that dashboard file is not valid JSON. Grafana
      skips it and keeps running, so only that dashboard is missing.
    - **`the same UID is used more than once`** or `dashboard title is not unique in
      folder`: two files define the same dashboard; only one wins.
    - **A datasource whose URL names a Service that does not exist** (or the wrong port):
      Grafana starts normally; every query fails, shown in the UI as "Bad Gateway" or a
      connection error, and in the log only when a query runs.
    - **`/api/health` returns `"database": "ok"`**: Grafana itself is up; `failing` (HTTP
      503) means its database is not reachable.
    
    State the cause in one sentence, then the evidence you saw (the log line, the ConfigMap
    entry, the Services), then the proposed fix.
    
    ## Proposed fix
    
    Show each fix to the user before running it: what it changes and how to undo it.
    Provisioning lives in ConfigMaps that come from a Helm chart, kube-prometheus-stack
    values or a GitOps repository: fix it there. Patching a ConfigMap that only holds
    provisioning or dashboards, and restarting Grafana to reload it, are reversible, so
    they go under "Ask first" and run only after the user says yes. Anything with Secrets
    (admin password, datasource credentials), Grafana's database, its volume and the
    replica count are never run by you: hand them to the user as proposed fixes. Write each
    undo with the values you found.
    
    Restart Grafana after its provisioning ConfigMap is fixed (Grafana reads datasource
    files only at start):
    
    > Ask first: the agent runs this only after the user approves it.
    
    ```bash
    kubectl rollout restart deployment/my-grafana -n monitoring
    # Undo: kubectl rollout undo deployment/my-grafana -n monitoring
    ```
    
    ## Sources
    
    Checked against the Grafana 13.2 documentation and source (v13.2.4), the Grafana Helm
    chart 14.0 and grafana-operator 5.25:
    
    - [Troubleshooting](https://grafana.com/docs/grafana/latest/troubleshooting/) and
      [Grafana on Kubernetes](https://grafana.com/docs/grafana/latest/setup-grafana/installation/kubernetes/).
    - [Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/):
      datasource and dashboard files, `isDefault`, providers.
    - [Prometheus datasource troubleshooting](https://grafana.com/docs/grafana/latest/datasources/prometheus/troubleshooting/).
    - [High availability](https://grafana.com/docs/grafana/latest/setup-grafana/set-up-for-high-availability/):
      a shared database for several replicas.
    - [Upgrade to Grafana 13](https://grafana.com/docs/grafana/latest/upgrade-guide/upgrade-v13.0/).
    - The [Grafana Helm chart](https://github.com/grafana-community/helm-charts/tree/main/charts/grafana)
      (sidecar labels, image, admin password) and the
      [grafana-operator documentation](https://grafana.github.io/grafana-operator/docs/).
    - Messages from the source:
      [provisioning.go](https://github.com/grafana/grafana/blob/v13.2.4/pkg/services/provisioning/provisioning.go),
      [datasources config_reader.go](https://github.com/grafana/grafana/blob/v13.2.4/pkg/services/provisioning/datasources/config_reader.go),
      [dashboards file_reader.go](https://github.com/grafana/grafana/blob/v13.2.4/pkg/services/provisioning/dashboards/file_reader.go).
      The messages quoted here were seen with Grafana 13.2.4 on a Kubernetes 1.37 kind
      cluster; the exact wording of the start-up error differs from older releases.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.