Observability Aiops

Disclaimer: Community-maintained open-source project. Not affiliated with, endorsed by, or sponsored by the Prometheus or Grafana projects, Grafana Labs, or the Cloud Native Computing Foundation. Prometheus, Alertmanager and Grafana are trademarks of their respective owners. MIT licensed.

Governed AI-ops for a self-hosted observability stack in one server — Prometheus (HTTP API, PromQL, targets, rules, alerts), Alertmanager (alerts + silences), Grafana (dashboards, datasources, folders), and Grafana Loki (bounded LogQL log reads + log RCA) — with a built-in governance harness: unified audit log, token/runaway budget guard, undo-token recording, and descriptive risk-tier labels. One config can span your whole stack; each target names its own platform. Beyond the mock test suite, the Prometheus/Alertmanager/Grafana reads, the RCAs, and the governed silence + dashboard write paths (with undo) have been exercised against a live Prometheus 3.x + Alertmanager + Grafana 13 stack — see docs/VERIFICATION.md.

This is the self-hosted-observability complement to enterprise monitoring suites: it speaks the open Prometheus/Grafana APIs an SRE actually runs, not a vendor NMS.

What it does

Answers the questions an SRE actually repeats over a Prometheus/Grafana stack, and guards the writes that follow:

• PromQL + metadata — instant and range queries, label-value enumeration, and series metadata, all read-only and result-capped.
• Scrape-target & rule health — which targets are up/down (and why, from lastError), which were dropped by relabeling, and which recording/alerting rules are erroring.
• Alerts & silences — firing/pending Prometheus rule alerts, Alertmanager's post-routing view, and its silences.
• Grafana — dashboards, datasources (+ health), and folders.
• Loki logs — bounded LogQL reads (label + label-value enumeration, a validation-gated query_range, and a canned error-tail), all read-only with a hard lookback + line cap, optional multi-tenant X-Scope-OrgID, and basic/bearer auth per target.
• Flagship analyses — transparent heuristics that show their numbers: firing_alert_rca (join each firing alert to its rule expr → cause + action), target_scrape_health_analysis (rank down/erroring scrapes → likely cause), alert_noise_and_flap_analysis (frequently-repeated / duplicate alerts → dedup/rollup recommendation), plus two log analyses — log_error_burst_rca (per-stream error burst vs baseline → new-signature / volume-spike / single-instance) and log_volume_analysis (top streams + high-cardinality label warnings + retention hint) — and alert_log_context, which correlates a firing Prometheus alert to its Loki streams.
• Governed writes — create/expire Alertmanager silences (time-boxed), create Grafana annotations, update/delete dashboards, and hot-reload the Prometheus config — each audited, risk-tiered, dry_run-able, and the reversible ones capture the real fetched before-state for undo.

What this tool does, and does not, decide

It delivers Prometheus + Grafana operations — reads and writes — accurately and efficiently, and records every one of them. It does not decide whether a write is allowed to happen. That is the agent's judgement, or the permission of the account you connect it with: give it a Grafana token with only Viewer scope, and a Prometheus/Alertmanager reached without the admin/write API, and the writes fail at the server — the place that actually owns the permission.

So there is no read-only switch, no policy file, no approval gate to configure. The one thing the tool guarantees is that nothing is silent: every call, over MCP and over the CLI alike, lands an audit row in ~/.observability-aiops/audit.db, and destructive writes still capture their before-state and record an inverse where one exists.