# Agent Evaluation Alert

## How to recognize this alert

| Signal | Value |
|---|---|
| `alert_shape` from `get_alert_agent_classification` | `agent_evaluation` |
| `incident_type` | `agent_evaluation_anomalies` |
| `event_type` | metric-shaped (e.g. `custom_metric_anom`) — NOT the discriminator for this type |
| `get_alerts` category (`alert_types`) | `"Agent evaluation"` |

`get_alert_agent_classification(alert_id)` is the authoritative check — it returns the
shape and the agent's `backend_class` in one call.

## What the alert means

An LLM judge scores the agent's outputs on a quality dimension, and the score fell below
(or spiked above) expected levels. Common judge fields: `helpfulness_score`,
`relevance_score`, `adherence_score`, `clarity_score`, `completion_score`,
`similarity_score`, `match_score`, `custom_eval_score`, plus boolean checks
(`content_safe`, custom pass/fail prompts) read as true/false rates, and rule-based
fields like `word_count`. Most numeric scores are on a 1–5 (or 0–1) scale.
`mismatch_score` is inverted — higher is worse.

The alert carries the breached metric and the breached field; the field names the judge
dimension you are investigating (e.g. `helpfulness_score`).

### Two grains — trace vs conversation

- **Trace/span grain (default):** each span or trace is judged individually. The
  breaching set is the traces of this agent, inside the alert's anomalous time
  bucket(s), matching the monitor's segment filter (e.g. a specific workflow or task).
- **Conversation grain:** the judge scores a whole multi-turn conversation as one unit.
  The alert payload does not carry conversation IDs — the breaching conversations are
  the evaluation-run samples on the BREACHING side of the score. Which side that is
  depends on the metric and breach direction (step 3) — it is NOT always the lowest
  scores.

> **CRITICAL:** Conversation-grain evaluation exists for agents on Monte Carlo's managed
> trace store (backend class `ao_clickhouse_otel`) and on the Snowflake Cortex /
> Databricks Genie platform backends. On the other backends (MLflow SDK/KA, customer
> OTel tables), evaluation alerts are span/trace grain — do not go looking for breaching
> conversations there.

To tell the grains apart: the monitor definition uses `*_conversation` judge variants
and conversation aggregation at conversation grain, and the monitor description usually
says so. If the backend is MLflow SDK/KA or a customer OTel table, it is span/trace
grain.

## Investigation playbook

1. **Classify and route.** `get_alert_agent_classification(alert_id)` → confirm
   `alert_shape` is `agent_evaluation` and read `agent.backend_class`. Open the matching
   backend reference before touching trace data.
2. **Pull the alert details** via `get_alerts`: the judge dimension, the threshold and
   direction, the anomalous time bucket(s), and any segment condition. The segment
   condition scopes everything that follows.
3. **Identify the breaching set** — the items on the BREACHING side of the score inside
   the breach window. Resolve the side from the metric + breach direction first:
   - Quality scores breaching low (the common case): the lowest-scoring items.
   - Quality scores breaching high (the score spiked above expected levels): the
     highest-scoring items.
   - Boolean/flag evals (`true_*`/`false_*` aggregations): the breaching items carry
     the value whose share ROSE — the metric's tracked value when breaching high, its
     complement when breaching low. `true` items sit at the TOP of the score range
     (score 1.0), `false` items at the BOTTOM (score 0.0). So `escalation_suggested`
     rising on a `true_count`/`true_rate` breaches at the TOP, while `content_safe`
     breaching on a rising `false_rate` breaches at the BOTTOM — "worst-scoring /
     bottom 10" selects exactly the wrong side for the former.
   - Inverted numerics (`mismatch_score`-style, breaching high): the highest scores.

   Then pull the items:

   - Trace grain: `get_agent_traces` filtered to the agent plus the alert's segment,
     within each anomalous bucket window.
   - Conversation grain: `get_agent_conversations` for the agent over the breach
     window; work from the ~10 most extreme conversations on the breaching side.
4. **Verify the items actually breach.**

   > **CRITICAL:** Sampling seams can hand you non-breaching items. Check every item's
   > score against the alert's threshold before treating it as evidence. A known failure
   > mode: a flag eval breaching HIGH on a true-count, sampled score-ascending
   > "worst-first" — its flagged rows sat at the TOP scores, so the page's limit cut
   > them off and seeded the investigation with perfectly-scoring conversations; the
   > investigator then "confirmed normal behavior" while reading the wrong conversations
   > entirely. Any sample sorted toward the non-breaching side (step 3) fails the same
   > way.

5. **Read the judge's scores and stored reasoning first.** The persisted judgment is the
   truth for this alert. The stored reasoning (often a paragraph per item) frequently
   names the failure mode outright — when it already explains the regression, capture it
   and stop drilling.
6. **Read the actual items.** `get_agent_conversation` per conversation (conversation
   grain) or `get_agent_trace` per trace (trace grain; managed-store (`ao_clickhouse_otel`)
   agents only — on other backends it errors, see the backend guide). Read the breaching items AND a
   few healthy items from before the breach began — the comparison is what isolates what
   changed. Note: raw content (prompts, completions, transcripts) is gated on the
   account's data-sampling consent; without it, reason from structure (span taxonomy,
   status, tokens, durations) and say so.
7. **Cluster the failure modes before concluding.** Group the breaching items by shared
   pattern — same workflow/task, same model, same kind of question, same failure shape
   (empty outputs, off-topic answers, refusals). One cluster with one cause is a
   different finding than three unrelated failures.

   > **NEVER** generalize from a single conversation or trace. A conclusion needs
   > multiple breaching items showing the same failure mode.

8. **Correlate with a change.** Evaluation breaches track the agent's *responses* — you
   do not see the data the agent ran on. Focus on prompt changes, model swaps,
   input-distribution shifts (a new kind of ask), and workflow adherence, comparing
   breaching items against pre-onset items. Record negative findings explicitly ("no
   prompt change detected").

The troubleshooting agent can run this alert end-to-end in parallel:
`run_troubleshooting_agent(incident_id)`, then `get_troubleshooting_agent_results` for its
evidence timeline. Merge rather than duplicate.

## Reading the results

- **Score direction matters.** Most judges: lower = worse. `mismatch_score`: higher =
  worse. Boolean checks read as rates (e.g. a rising false rate on `content_safe`).
- **Platform backends (Databricks Genie, MLflow-based agents):** the evaluation score is
  computed by Monte Carlo and lives on the alert/monitor — it is NOT a column in the
  trace data. The traces tell you what the agent *did*; the monitor tells you what
  *scored* low. Do not hunt for a score field in span data.
- **Snowflake Cortex:** evaluations run inside the customer's warehouse, so a quality
  movement correlates with an agent-configuration change or a source-data shift — not
  with a Monte Carlo scoring change.
- A quality drop is the symptom (`quality regression`); the finding is complete only
  when paired with the causal change (prompt edit, model swap, config change, new input
  mix) or an explicit "no correlated change found".

## Common mistakes

| Mistake | Why it fails / what to do instead |
|---|---|
| Assuming breaching = lowest scores | The breaching side follows metric + direction (step 3): a flag eval breaching high on a true-count breaches at the TOP scores; a rising `false_rate` (e.g. `content_safe`) breaches at the BOTTOM — resolve the side before sampling |
| Concluding from one conversation | Cluster several breaching items; one item proves nothing about the population |
| Treating sampled items as breaching without checking scores | Sampling seams return non-breaching items; verify each score against the threshold |
| Expecting conversation grain on MLflow or customer-OTel backends | Conversation-grain evaluation runs on the managed store and Cortex/Genie only — elsewhere alerts are span/trace grain |
| Selecting an eval score from trace data on Genie/MLflow | The score lives on the alert/monitor, not in the spans |
| Misreading `mismatch_score` | It is inverted — higher is worse |
| Only reading breaching items | Always compare against pre-onset healthy items to isolate what changed |
| Silent dead ends | Record negative findings ("no model change in window") — they narrow the cause |

## Related references

- How this monitor type is defined:
  `../../monitoring-advisor/references/agent-evaluation-monitor.md`
- Backend-specific signal and gotchas: the `agent-backend-*.md` file the router selected.

# Agent Metric Alert

## How to recognize this alert

| Signal | Value |
|---|---|
| `alert_shape` from `get_alert_agent_classification` | `agent_metric` |
| `incident_type` | `agent_metric_anomalies` — NOT the generic (non-agent) metric anomaly type |
| `event_type` | metric-shaped — NOT the discriminator for this type |
| `get_alerts` category (`alert_types`) | `"Agent metric"` |

`get_alert_agent_classification(alert_id)` is the authoritative check — it returns the
shape and the agent's `backend_class` in one call.

## What the alert means

A built-in quantitative span metric moved outside its expected range — no LLM judging
involved. The metrics: latency (`duration_sec`), token counts (`prompt_tokens`,
`completion_tokens`, `total_tokens`), LLM-call counts, error rate (span status), and
trace volume. A breach signals a **performance or cost regression** — a latency spike,
a token explosion, an elevated error rate, a volume cliff — not a quality degradation.

The alert names the monitor, the breached metric, the anomalous time bucket(s), and any
segment condition (e.g. `task = 'summarize'`). The segment condition is part of the
alert's meaning: the regression was detected *inside that segment*, and the
investigation should start scoped to it.

## Investigation playbook

1. **Classify and route.** `get_alert_agent_classification(alert_id)` → confirm
   `alert_shape` is `agent_metric` and read `agent.backend_class`. Open the matching
   backend reference — it decides which of these signals even exist (see step 6's token
   caveat).
2. **Pin down what breached.** From the alert details (`get_alerts`): which monitor,
   which metric, which direction, which segment, which time bucket(s).
3. **Trend vs baseline.** `get_agent_traces` over the breach window AND over a
   comparable prior window (equal length immediately before; a 7-day lookback works
   well). Establish the metric's day-by-day shape and find the **onset date** — and
   whether it is a *step* (discrete change landed that day) or a *drift* (gradual
   growth).

   > **CRITICAL:** An anomaly is defined by what CHANGED, not by the state of the bad
   > window alone. Never describe only the bad window — always ground it against the
   > baseline.

4. **Segment isolation — find WHERE before asking why.** `get_agent_segments` to
   enumerate the agent's workflows, tasks, and models, then filtered `get_agent_traces`
   per candidate segment. A regression confined to one workflow, one task node, or one
   model is a different root cause than a fleet-wide one. Also check the complement of
   the alert's segment: is the rest of the agent healthy?
5. **Error correlation.** Did error counts move together with the metric? Distinguish
   provider rejections (LLM span fails fast with no output and no tokens), timeouts
   (unusually long failing spans), and code errors (exception text). An error spike that
   coincides with a latency spike usually shares its cause.
6. **Correlate with changes.** The usual suspects, checked against the onset date:
   - a **model swap** (a new model appearing on a node at the onset — different context
     window, throughput, or pricing behavior);
   - **prompt size growth** or per-trace context accumulation (message arrays growing
     turn over turn until token counts blow up);
   - a **configuration or prompt change** landing at the onset;
   - a **code change / deploy** — for code agents only, and only changes that landed
     BEFORE the onset (a change merged after the issue started cannot be its cause;
     allow margin for deploy lag);
   - a **provider-side incident** — a sharply time-bounded spike across many traces
     points at the provider; check its public status history for the onset date.
7. **Drill into exemplar traces.** `get_agent_trace` on two or three of the worst
   traces from the breach window and one or two healthy traces from before the onset.
   Compare span by span: where does the time go, where do the tokens go, which span
   grew or started failing. (`get_agent_trace` reads managed-store (`ao_clickhouse_otel`)
   agents only — on other backends it errors; get span-grain depth from
   `run_troubleshooting_agent`, per the backend guide.)

## Reading the results

- **Step vs drift is the first fork.** A step change points at a discrete change on
  that date (model, prompt, config, deploy). A drift points at accumulation — growing
  inputs, growing context, growing data volume.
- **Segment-confined vs fleet-wide.** Confined to one node/model → look at that
  component's change history. Fleet-wide → look at shared infrastructure, the provider,
  or a global config change.
- **Token blowups:** find the span where accumulation begins — a late-stage overflow is
  often caused by earlier stages growing the context.
- **Slow but healthy is a risk, not an error.** Keep latency outliers separate from
  failures in your evidence.
- **Tokens do not exist on every backend.** Databricks Genie and Knowledge Assistant
  agents record no model and no token counts — token, cost, and model-swap findings are
  invalid there. The backend reference states what exists.

## Common mistakes

| Mistake | Why it fails / what to do instead |
|---|---|
| Describing only the bad window | Always compare against a baseline window; the finding is the *change* |
| Skipping segment isolation | A one-node regression blamed on the whole fleet (or vice versa) misdirects the fix |
| Blaming a change that landed after the onset | Only changes before the onset can be causal; allow deploy-lag margin |
| Token/cost/model findings on Genie or Knowledge Assistant | Those backends have no tokens or model data — the finding is fabricated |
| Counting slow-but-successful traces as errors | Latency risk and failures are different evidence; keep them separate |
| Ignoring the alert's segment condition | The breach was detected inside that segment; investigate there first, then check the complement |

## Related references

- How this monitor type is defined:
  `../../monitoring-advisor/references/agent-metric-monitor.md`
- Backend-specific signal and gotchas: the `agent-backend-*.md` file the router selected.

# Agent Trajectory Alert

## How to recognize this alert

| Signal | Value |
|---|---|
| `alert_shape` from `get_alert_agent_classification` | `agent_trajectory` |
| `incident_type` | `custom_rule_anomalies` (shared with plain custom rules — not sufficient alone) |
| `event_type` | `agent_trajectory_anom` — the discriminator |
| `get_alerts` category (`alert_types`) | `"Agent trajectory"` |

Trajectory alerts arrive as **custom-rule alerts**: the payload carries the rule
definition and a hit *count* — there is no monitor aggregation bucket and, importantly,
**no list of offending trace IDs**.

## What the alert means

A rule asserted something about the *execution shape* of each trace, and one or more
traces matched the violating pattern. Rules combine two kinds of assertion (AND/OR):

- **Occurrence:** span X must occur more than / fewer than / exactly N times per trace —
  catches runaway loops, excessive LLM or tool calls, and missing steps.
- **Order/relation:** span A must occur before / after / together with (or never with)
  spans B, C — catches skipped steps, reordered flows, and forbidden combinations.

A trajectory violation is a *pattern*, not an error: the violating traces are often
status-healthy. It usually indicates a **control-flow regression** — the agent's
decision path changed.

> **CRITICAL:** The rule's own selection logic — the query that identifies exactly which
> traces violated — is NOT retrievable through the toolkit tools. You either take the
> exact set from the troubleshooting agent's results, or approximate it from the rule's
> described intent.

## Investigation playbook

1. **Classify and route.** `get_alert_agent_classification(alert_id)` → confirm
   `alert_shape` is `agent_trajectory` and read `agent.backend_class`. Open the matching
   backend reference.
2. **Read the rule's intent** from the alert title and description (via `get_alerts`):
   which span(s), which count or ordering assertion, over which window. Write it down as
   a plain sentence — "traces where `web_search` ran more than 15 times" — before
   querying anything.
3. **Prefer the exactly-resolved violating set.** Call
   `get_troubleshooting_agent_results(incident_id)`: when the troubleshooting agent has run
   on this alert, its findings contain the exactly-resolved violating trace IDs — the
   set the rule actually matched. Merge your investigation with those traces rather than
   re-deriving a cohort. If it has not run, kick it off with
   `run_troubleshooting_agent(incident_id)` and continue manually in parallel.
4. **Otherwise approximate candidates.** `get_agent_traces` over the alert window,
   filtered by whatever proxy signal the rule implies: LLM-call counts (runaway loops),
   trace duration, error counts, workflow/task. For "span X more than N times", find the
   traces with the highest call counts; for a missing-step rule, pull traces of the
   affected workflow and check their span trees.
5. **Anchor the window to the alert's own timestamp.**

   > **CRITICAL:** Trajectory rules typically evaluate "the last N hours *as of when the
   > rule runs*". Replaying the same logic later selects a different set of traces.
   > Search `[alert time − rule window, alert time]` — never "the last N hours from
   > now".

6. **Diff violating vs compliant.** `get_agent_trace` on candidate violating traces AND
   on a compliant trace from the same workflow (managed-store (`ao_clickhouse_otel`)
   agents only — on other backends it errors; get the span-shape view from
   `run_troubleshooting_agent`, per the backend guide). Answer concretely: WHICH
   expected span is missing, out of order, or repeated too many/few times?
7. **Commonality and onset.** What do the violating traces share (workflow, task, input
   kind, user) that compliant traces don't? Since WHEN do violations appear — compare
   against earlier traces of the same workflow to date the onset.
8. **Correlate with a change.** Trajectory violations are usually control-flow
   regressions, so weight code/deploy and prompt/routing changes heavily: a prompt edit
   that reordered or dropped a step, a new branch that skips a tool, a dependency or
   configuration change, or an exception that aborts the expected step (check for errors
   inside or just before the missing span).

## Reading the results

- **Healthy status ≠ compliant trajectory.** Do not filter candidates to errored traces;
  the violation lives in the span tree's shape.
- **Repeated spans often wrap a failure.** A loop violation is frequently a retry loop
  around a silently failing call — check the repeated span (and its children) for
  errors and identical inputs.
- **You see the spans the agent emitted, not the data it processed.** Diagnose the
  missing/reordered/repeated step from the span tree; the *reason* the agent took that
  path usually needs the prompt/routing change correlation from step 8.
- Describe the specific deviation in your finding ("the validation step stopped running
  after the router prompt change on the 14th"), not just "the rule fired".

## Common mistakes

| Mistake | Why it fails / what to do instead |
|---|---|
| Searching "recent" traces | The rule's window is anchored to run time; re-anchor to `[alert time − window, alert time]` |
| Expecting offending trace IDs in the alert | The payload carries only the rule and a hit count; resolve traces via the troubleshooting agent or approximation |
| Re-deriving the cohort when troubleshooting-agent results exist | `get_troubleshooting_agent_results` already has the exactly-resolved set — merge with it |
| Filtering candidates to errored traces | Trajectory violations are patterns; violating traces are often status-healthy |
| Concluding from one violating trace | Always diff against a compliant trace of the same workflow, and check several violators for commonality |
| Ignoring code/prompt/routing changes | Control-flow regressions almost always trace back to one; weight them heavily |

## Related references

- How this monitor type is defined:
  `../../monitoring-advisor/references/agent-trajectory-monitor.md`
- Backend-specific signal and gotchas: the `agent-backend-*.md` file the router selected.

# Agent Validation Alert

## How to recognize this alert

| Signal | Value |
|---|---|
| `alert_shape` from `get_alert_agent_classification` | `agent_validation` |
| `incident_type` | `custom_rule_anomalies` (shared with plain custom rules — not sufficient alone) |
| `event_type` | `agent_validation_anom` — the discriminator |
| `get_alerts` category (`alert_types`) | `"Agent validation"` |

Same custom-rule wire shape as trajectory alerts: the payload carries the rule
definition — there is no monitor aggregation bucket and **no list of offending trace
IDs**.

## What the alert means

A logical assertion over individual span fields matched one or more spans the monitor
considers INVALID. Typical assertions:

- **Numeric ceilings/floors:** total tokens per span must stay under N; duration under
  a limit.
- **Non-null / presence requirements:** a field the pipeline depends on must be
  populated.
- **Compliance / content conditions:** the output must (or must not) contain something.
- **Hard-failure conditions:** a given tool or LLM span must not error.

The alert fires when at least one span matches the rule in its run window — by default
the rule looks back **about one hour** from each run, not a whole day.

> **CRITICAL:** The rule's own selection logic — the query that identifies exactly which
> spans violated — is NOT retrievable through the toolkit tools. The exact violating
> rows live in the troubleshooting agent's results; otherwise you approximate them from
> the assertion described in the alert.

## Investigation playbook

1. **Classify and route.** `get_alert_agent_classification(alert_id)` → confirm
   `alert_shape` is `agent_validation` and read `agent.backend_class`. Open the matching
   backend reference.
2. **Read the assertion** from the alert title and description (via `get_alerts`): what
   condition marks a span invalid? Then classify it — it decides your drill-in:
   - a **hard failure** (a tool/LLM span erroring) → start from the failed spans;
   - a **content assertion** (output must/must not contain something) → you will need
     to read the breaching spans' content;
   - a **numeric ceiling** (tokens, duration) → treat like a targeted metric check.
3. **Prefer the exactly-resolved rows.** Call
   `get_troubleshooting_agent_results(incident_id)`: when the troubleshooting agent has run,
   its findings contain the exactly-resolved breaching traces/spans — merge with those
   rather than re-deriving. If it has not run, kick it off with
   `run_troubleshooting_agent(incident_id)` and continue manually in parallel.
4. **Otherwise find the offending traces/spans.** `get_agent_traces` filtered by the
   assertion's signal — error status, span name, workflow/task, token or duration
   thresholds — over the window ending at the alert's timestamp (default lookback about
   one hour). Anchor to the alert's own time, not to "now".
5. **Identify the span-level cause.** `get_agent_trace` per offending trace
   (managed-store (`ao_clickhouse_otel`) agents only — on other backends it errors;
   get span-grain detail from `run_troubleshooting_agent`, per the backend guide):
   - Hard failure: locate the failed spans in the tree; in a cascade of failures the
     root cause is usually the **earliest or innermost** failing span.
   - Content assertion: read what the breaching spans' outputs actually contain that
     trips the rule (raw content is gated on the account's data-sampling consent;
     without it, reason from structure and say so).
   - Numeric ceiling: find which span carries the excess and whether it grew over time.
6. **Commonality.** Which spans violated, and what do they share — same workflow, task,
   model, tool, time window, or input shape? A single tool failing everywhere is a
   different story than everything failing in one workflow.
7. **What changed.** Compare the violating spans against comparable spans from before
   the breach began: a code/tool regression, a prompt or configuration change, a model
   swap, or a provider-side failure. The assertion tells you WHAT is invalid; the
   before/after comparison tells you WHY it started.

## Reading the results

- **Failure signatures on LLM spans:** an error with no completion and no token counts
  usually means the provider rejected the call; a very short failing span is a fast
  fail (bad request, auth, config); an unusually long one is a timeout.
- **Error-like text inside a span's inputs is NOT a failure.** Prompts routinely quote
  exception text as context. Trust the span's status, never string-matching inside
  content — healthy spans often *mention* more errors than failing ones.
- **Cascades:** multiple failing spans in one trace generally share one root cause —
  work from the earliest/innermost failure outward.
- A content assertion that newly fails is itself a quality regression; the change that
  introduced it (prompt, config, model, code) is the separate causal finding.

## Common mistakes

| Mistake | Why it fails / what to do instead |
|---|---|
| Investigating whole-day windows | The rule looks back ~1 hour from each run; scope to `[alert time − lookback, alert time]` |
| Treating quoted error text in inputs as failures | Status decides; content routinely quotes exceptions as context |
| Re-deriving rows when troubleshooting-agent results exist | `get_troubleshooting_agent_results` has the exactly-resolved violating rows — merge with them |
| Using the wrong drill-in for the assertion kind | Hard failure → failed spans first; content assertion → span content; don't swap them |
| Expecting offending trace IDs in the alert | The payload carries only the rule; resolve rows via the troubleshooting agent or approximation |
| Stopping at "the assertion failed" | Pair the invalid spans with the before/after change that made them start failing |

## Related references

- How this monitor type is defined:
  `../../monitoring-advisor/references/agent-validation-monitor.md`
- Backend-specific signal and gotchas: the `agent-backend-*.md` file the router selected.

# Backend: Monte Carlo OTel Store (`ao_clickhouse_otel`)

## What this backend is

The Monte Carlo–managed trace store for OpenTelemetry-instrumented **code agents**. The
customer instruments their own agent code with OTel and Monte Carlo ingests the spans.
This is the richest backend: everything the platform can know about an agent run exists
here, so the investigation is limited by your discipline, not by the data.

**CRITICAL: there is no built-in previous-period baseline on this backend — every trend
you read describes only the window you asked for. Always construct your own comparison
window (e.g. the 7 days before the onset vs the window after it) before calling anything
a regression.**

## Signal available here

- **Full, real span tree** — parent/child structure, per-span timing, and error flags via
  `get_agent_trace`.
- **Model per span and token counts** (prompt / completion / total) — model-swap, cost,
  and context-overflow questions all have answers here.
- **Error detail** — status per span plus error type and error message text.
- **Workflow / task / model segmentation** — enumerate real segment values with
  `get_agent_segments`; group and sort traces with `get_agent_traces`.
- **Conversations** — `get_agent_conversations` / `get_agent_conversation`, with
  transcripts when the account's data-sampling settings allow content.
  **Conversation-grain evaluation monitors** run here (and on the Snowflake Cortex and
  Databricks Genie platform backends), so a breached eval alert may name whole
  conversations rather than traces.
- **Conversation clustering** — when the account has clustering enabled for this agent,
  Monte Carlo groups its conversations into an intent-cluster taxonomy, shown alongside
  the agent's conversations in the Monte Carlo UI. No toolkit tool reads clusters
  directly: point the user at the cluster view to see which *kind* of conversations a
  regression concentrates in, or hand off to `run_troubleshooting_agent`, which uses
  cluster-share shifts as evidence.
- **Change correlation** — this is a code agent: when the customer has GitHub connected,
  correlate the onset with merged PRs and deploys.

## Absent by design — do not chase

- **No previous-period deltas** — trend reads are single-window; the baseline is yours to
  build.
- **No platform config surface** — behavior changes land through code deploys, not a
  declarative agent configuration. "What config changed?" is answered by PR/deploy
  history here, not by an agent settings diff.

## Investigation approach

1. **Confirm the backend first** — `get_alert_agent_classification` for an alert, or
   `get_agent_metadata` for the agent. The backend decides what signal exists and what
   the fix language is.
2. **Establish trends and find the onset.** Over a window of roughly 7 days before the
   earliest anomalous trace to 1 day after the latest, reproduce the trend dimensions
   with `get_agent_traces` aggregation (sort/group by the relevant field). The
   dimensions worth reproducing:
   - latency percentiles per node/task (typical vs tail)
   - error rate per node/task
   - throughput (trace and span volume over time)
   - token usage (prompt / completion / total)
   - prompt stability (did prompt sizes change, or prompts disappear, on a date?)
   - error-type breakdown over time
   - prompt/completion length growth
   - per-trace context accumulation (message counts growing span-over-span inside a
     trace — `get_agent_trace` on representative traces)
   Decide whether the movement is a **step** (look for a discrete change on that date)
   or a **drift**.
3. **Classify errors before hypothesizing.** Distinguish: provider rejection (LLM span
   errored with no tokens and no completion), timeout (far above the node's typical
   latency), fast-fail (errored in seconds), structured-output/parse failure, code
   exception, and slow-but-healthy (a risk, not an error).
4. **On a known-bad trace, list its failed spans in order** with `get_agent_trace`.
   Cascade failures show as multiple failed spans — the root cause is usually the
   earliest or innermost one. Read the actual error text before forming a hypothesis.
5. **Compare content across cohorts** (when content is available): read breaching
   conversations/traces and pre-onset ones, and compare — prompt changes, empty outputs,
   and shared failure patterns show up here.
6. **Correlate with changes.** PRs merged **before** the onset (a PR merged after the
   issue started cannot be the root cause; merges can precede deploys, so use wide
   margins), provider status pages for time-bounded spikes, and model changelogs for
   context-window or behavior changes.
7. Named plays worth knowing: **model switch** (non-overlapping model date ranges on the
   same node → before/after error pivot → check the new model's context window and
   whether prompts stayed identical); **provider rejection** (token/prompt growth
   upstream? new model at onset? time-bounded across traces ⇒ provider outage);
   **context overflow** (tokens usually present but now missing on failures ⇒ provider
   failure or overflow; overflow in a late node ⇒ the accumulation began in an earlier
   stage).
8. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
   collect the evidence timeline with `get_troubleshooting_agent_results`.

## Gotchas

- **Error status is effectively binary** — a span is either an error or it is healthy;
  do not invent intermediate severities from message text.
- **Interrupts are not errors.** LangGraph-style "interrupt" messages are control flow —
  exclude them from any error-rate reasoning.
- **Exception text quoted inside a span's input context is not a real error.** Healthy
  spans often *mention* more exceptions than failing ones (they carry prior errors as
  context). Trust the span's error status, never text matching on content.
- **Generic node names repeat.** A name like `RunnableSequence` can appear at several
  places in the graph — disambiguate by the span's position in the tree
  (`get_agent_trace` parent path), not by name alone.
- **Compare prompts by identity, not by prose.** Whether prompts are identical or
  changed across cohorts matters more than what they say — length and sameness are the
  first-class signals.
- **Units differ across tools.** `get_agent_traces` reports `duration_seconds`;
  `get_agent_trace` reports duration in milliseconds; monitors use `duration_sec`. See
  the span-field reference before quoting numbers.

## Cross-links

- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Alert-type playbooks: `agent-alert-*.md`

# Backend: Snowflake Cortex / Snowflake Intelligence (`platform_agent`)

## What this backend is

A **declarative platform agent** running inside the customer's Snowflake account —
Cortex Agents / Snowflake Intelligence. There is no agent source code: the agent is
defined by its configuration (instructions, tools, semantic models/views), and Snowflake
runs it. Monte Carlo reads its AI-observability events, normalized to the standard span
vocabulary.

**CRITICAL: there is no code repo and no PRs here — the fix surface is the agent's
configuration. A "which PR broke it" question has no answer on this backend; a "which
config change broke it" question usually does.**

## Signal available here

- **Real span structure with token counts** — token, duration, and status questions all
  have answers; latency and error trends are readable through `get_agent_traces`.
- **A config surface with change timestamps** — the agent's instructions, tools, and
  semantic views carry created/modified times. This is the platform analog of code
  history and the primary change-correlation signal.
- **Conversations** — `get_agent_conversations` (rank by tokens/duration/errors to find
  the expensive or failing ones) and `get_agent_conversation` for a full thread.
  Transcript content is consent-gated (see Gotchas). **Conversation-grain evaluation
  monitors** run on this backend, so a breached eval alert may name whole conversations.
- **Conversation clustering** — when the account has clustering enabled for this agent,
  Monte Carlo groups its conversations into an intent-cluster taxonomy, shown alongside
  the agent's conversations in the Monte Carlo UI. No toolkit tool reads clusters
  directly: point the user at the cluster view to see which *kind* of conversations a
  regression concentrates in, or hand off to `run_troubleshooting_agent`, which uses
  cluster-share shifts as evidence.
- **Data lineage bridge** — the agent's generated SQL queries real source tables; a
  freshness/volume/schema incident on a source table (check `get_alerts`) is a data
  root cause only Monte Carlo can surface.
- **Segments** — `get_agent_segments` for the workflow/task values in play.

## Absent by design — do not chase

- **No source code, no GitHub, no PRs.** Do not look for repositories or recommend code
  changes.
- **No OTel exception attributes** — error semantics come from span status and the
  tool-span structure, not from exception type/message fields.
- **Latency and infrastructure are Snowflake-managed** and not user-fixable — focus on
  answer quality and cost via the config surface, not on infra tuning.
- **Raw content is consent-gated** — prompts, completions, generated SQL, and
  transcripts require the account's data-sampling consent. Structure, status, tokens,
  and config are always available.

## Investigation approach

1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`.
2. **Config-space investigation FIRST.** Compare the agent's configuration modification
   times (instructions, semantic views, tools) against the regression window. An
   instruction edit, a semantic-model change, or a tool change landing right before the
   anomaly is a top root cause on this backend.
3. **Compare BEFORE vs AFTER.** The incident window vs the immediately-prior baseline —
   traces, tokens, duration, errors — plus the longer daily trend via `get_agent_traces`
   aggregation. Did the breached metric *step* on a specific day (points at a config
   change that day) or drift? Never describe only the bad window.
4. **Look at WHAT breached — and rule out a false positive.** Pull the flagged
   conversations (`get_agent_conversations`, sampled from the breaching side of the score
   per the metric + breach direction — see `agent-alert-evaluation.md` step 3) and read
   them (`get_agent_conversation`). Decide: genuine problem (runaway tool loop,
   bloated answer, error spike) vs false positive (a legitimately long-but-correct
   conversation, an expected seasonal spike, a too-tight threshold). A breach the sample
   shows to be benign **is a finding** — say so and recommend adjusting the monitor.
   When clustering is enabled for the agent, localize first: a cluster whose share moved
   in the breach window tells you which kind of conversations to sample (cluster view in
   the UI, or the automated run's cluster evidence).
5. **Bridge to the data.** Identify the source tables the agent queried and check
   `get_alerts` for incidents on them — a data incident upstream explains a quality drop
   better than anything in the agent itself.
6. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
   collect results with `get_troubleshooting_agent_results`.

## Gotchas

- **Consent gating is a fact to report, not a failure.** If conversation content comes
  back blocked or empty because of the account's data-sampling settings, say that content
  is unavailable on this account and continue from structure, tokens, and config — do
  not retry or treat it as an error.
- **Prefer summaries and rollups over raw span pulls.** Wide raw reads get truncated on
  this backend; aggregate views (`get_agent_traces` grouping, `get_agent_segments`,
  conversation lists ordered by impact) are the reliable way to see the picture.
- **Eval scores are computed in-warehouse.** A quality-score movement correlates with a
  config-surface change or a source-data shift — never with a Monte Carlo–side scoring
  change.
- **Schema questions are answered from the normalized vocabulary.** Do not run
  exploratory queries against the trace store to discover fields — the field list is in
  the span-field reference below.
- **Fix language is config-surface only:** edit instructions, adjust semantic views,
  add/remove tools, fix the upstream data incident. Never recommend code changes or
  infrastructure tuning.

## Cross-links

- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Alert-type playbooks: `agent-alert-*.md`

# Backend: Customer Snowflake OTel Table (`customer_otel_trace_table`)

## What this backend is

A **customer-owned Snowflake table** holding raw OpenTelemetry export from a
code agent the customer instrumented themselves. Monte Carlo normalizes it to the same
standard span vocabulary as the managed OTel store, so the standard list/aggregate reads
(`get_agent_traces`, conversations, segments) all work the same way. The exception is
`get_agent_trace`: the single-trace span-tree read resolves only against the Monte
Carlo–managed OTel store, so it errors on this backend — span-grain depth comes from
`run_troubleshooting_agent` instead. Investigation shape is essentially the managed-OTel
playbook — what differs is where the data lives and who feeds it.

**CRITICAL: the trace table is the customer's — ingestion gaps on their side (a stalled
export job, missing hours, a frozen latest-timestamp) can masquerade as agent
regressions. Rule out a feed gap before calling anything an agent problem.**

## Signal available here

- **Full, real span tree** with parent/child structure, timing, and error detail
  (including exception type/message) — captured in the normalized data. There is no
  direct MCP span-tree read here (`get_agent_trace` is managed-store-only): span-grain
  drill-down goes through `run_troubleshooting_agent`, which queries the trace table
  server-side.
- **Model per span and token counts** — model-swap, cost, and context-overflow questions
  have answers.
- **Workflow / task / model segmentation** — `get_agent_segments`, `get_agent_traces`.
- **Conversations** — `get_agent_conversations` / `get_agent_conversation`; transcript
  content is consent-gated.
- **Change correlation** — this is a code agent: when the customer has GitHub connected,
  correlate the onset with merged PRs and deploys.

## Absent by design — do not chase

- **Nothing structural vs the managed OTel store in the data** — the normalized
  vocabulary is the same; the difference is the store, not the signal. (Tooling does
  differ in one way: `get_agent_trace` reads only the managed store — see above.)
- **Conversation-grain eval breaches and conversation clustering** exist only on the
  Monte Carlo–managed store and the Cortex/Genie platform backends — eval alerts here
  resolve at trace/span grain.
- **No built-in previous-period baseline** — construct your own before/after comparison
  window, exactly as on the managed store.

## Investigation approach

1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`.
2. **Check the feed before the agent.** Look at trace volume over time with
   `get_agent_traces`: does the data simply stop, gap, or go stale around the anomaly?
   A completeness/freshness problem in the customer's export pipeline explains a
   "regression" without any agent change — and is itself the finding.
3. **Follow the managed-OTel playbook** (see `agent-backend-clickhouse.md`): establish
   the trend dimensions over ~7 days before the onset to 1 day after, find the onset,
   decide step vs drift.
4. **Classify errors before hypothesizing** — provider rejection, timeout, fast-fail,
   parse failure, code exception, slow-but-healthy.
5. **On a known-bad trace**, get the failed-spans-in-order view from the automated run
   (`run_troubleshooting_agent` — `get_agent_trace` errors on this backend); the root
   cause is usually the earliest or innermost failure. Read the actual error text.
   Manual MCP reads stop at trace grain (`get_agent_traces` — per-trace status and
   error counts).
6. **Compare content across cohorts** when the account's data-sampling settings allow —
   breaching vs pre-onset.
7. **Correlate with changes** — PRs merged before the onset (wide margins for deploy
   lag), provider status pages, model changelogs.
8. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
   collect results with `get_troubleshooting_agent_results`.

## Gotchas

- **Anchor the window on the alert's own time, not on "now".** The customer table is
  historical data — investigating an older incident with a "recent window" assumption
  finds nothing. Build the window around the incident timestamp.
- **A sudden drop to zero traces is a feed problem until proven otherwise** — treat
  volume cliffs and frozen timestamps as ingestion candidates first.
- **Schema questions are answered from the normalized vocabulary.** The raw table's
  own columns are not the fields you read — never explore the raw table to discover
  fields; use the span-field reference below.
- **Consent gating is a fact to report, not a failure** — if transcript content is
  blocked by the account's data-sampling settings, say so and reason from structure.
- **PR evidence is valid and high-value here** — this is a code agent; do not switch to
  config-surface framing.
- All the managed-OTel gotchas apply: interrupts are not errors, exception text quoted
  in input context is not a real error, generic node names need tree-position
  disambiguation, and a PR merged after onset cannot be the root cause.

## Cross-links

- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Managed-store playbook: `agent-backend-clickhouse.md`
- Alert-type playbooks: `agent-alert-*.md`

# Backend: Databricks Genie (`databricks_genie`)

## What this backend is

A Databricks **Genie space** — a declarative NL2SQL agent. Users ask natural-language
questions; Genie generates and runs SQL against curated tables. This is the **coarsest
signal** of any backend: one record per turn, with the generated SQL as the only "tool
call". The investigation is SQL-and-lineage-centric, not span-tree-centric.

**CRITICAL: model and token fields are empty by design here — a token-usage, cost, or
model-swap question has no answer on this backend. Do not produce token/cost/model
findings.**

## Signal available here

- **Turn-grain records** shaped as a shallow two-level tree: a root turn span (the NL
  question and the answer) with one child per generated SQL statement — the whole story
  for a turn. Read turns through the conversation tools (`get_agent_trace` does not
  read this backend — it is a managed-store-only tool and errors here).
- **Native conversation grouping** — every span carries a real conversation id;
  `get_agent_conversations` / `get_agent_conversation` reconstruct multi-turn threads.
  **Conversation-grain evaluation monitors** run on this backend, so a breached eval
  alert may name whole conversations.
- **Conversation clustering** — when the account has clustering enabled for this space,
  Monte Carlo groups its conversations into an intent-cluster taxonomy, shown alongside
  the space's conversations in the Monte Carlo UI. No toolkit tool reads clusters
  directly: point the user at the cluster view to see which *kind* of questions a
  regression concentrates in, or hand off to `run_troubleshooting_agent`, which uses
  cluster-share shifts as evidence.
- **Per-turn failure status**, and for failed turns the **recorded Genie error** (a real
  error type and message captured by the collector) surfaced in the span's attributes —
  not just a generic failure wrapper.
- **The generated SQL itself** (consent-gated content) — what questions were asked, what
  SQL Genie wrote, whether that SQL failed or changed shape.
- **Volume and failure trends** — turns, conversations, failed turns, duration, via
  `get_agent_traces` aggregation.
- **A config surface** — the space's instructions, curated/annotated tables, and example
  SQL / benchmark questions are the platform analog of code history.

## Absent by design — do not chase

- **No model, no tokens** — always null. No cost or "expensive" framing applies.
- **No real span tree** — the two-level tree is fabricated for presentation. Do not
  analyze intra-trace execution structure, span ordering, or nesting depth.
- **No agent source code and no PRs** — a Genie space is declarative.
- **Eval scores are not in the spans** — quality scores are Monte Carlo–computed and
  live on the monitor/alert. The spans tell you what a turn *did*; the alert tells you
  what *scored* low. Don't hunt for a score field in trace data.
- **Private conversations may not be ingested** — reason over the ingested slice (the
  same slice the monitor evaluated), not necessarily every conversation in the space.

## Investigation approach

1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`.
2. **Run the lineage play FIRST (the primary root-cause move).** Identify the source
   tables the space's generated SQL queried, then check `get_alerts` for
   freshness/volume/schema incidents on those tables. An upstream data incident is the
   single most likely root cause of a wrong, empty, or failed Genie answer — and the
   bridge only Monte Carlo can draw. Lead with this.
3. **Compare BEFORE vs AFTER.** Incident window vs the prior baseline: turns,
   conversations, generated-SQL volume, failed turns, duration — **not tokens** — plus
   the daily activity trend. A step on a specific day points at a curated-table or
   instruction change that day.
4. **Look at WHAT happened — and rule out a false positive.** Start content-free: list
   conversations with turn/failure counts (`get_agent_conversations`), then drill into a
   flagged conversation's question / answer / generated SQL with
   `get_agent_conversation` (content is consent-gated). Genuine problem (wrong NL2SQL
   translation, unfiltered scan, error spike) vs false positive (a legitimately hard
   question, an expected spike, a too-tight threshold). When clustering is enabled for
   the space, localize first: a cluster whose share moved in the breach window tells
   you which kind of questions to sample (cluster view in the UI, or the automated
   run's cluster evidence).
5. **For FAILED turns, read the recorded error before hypothesizing.** The recorded
   error type/message in the failed turn's span attributes IS the literal root-cause
   signal (e.g. a schema-access error). Fall back to structural reasoning only when no
   recorded message exists (older collector installs).
6. **Diff the config surface** — instructions, curated/annotated tables and their column
   descriptions, example SQL — against the onset date.
7. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
   collect results with `get_troubleshooting_agent_results`.

## Gotchas

- **The recorded error beats the wrapper.** A failed turn may show a generic
  "run failed" label; the real recorded error type and text live in the span's
  attributes — always read those specific attributes, never settle for the wrapper.
- **Content is JSON-string shaped.** Prompt/completion content on this backend is stored
  as JSON strings, not structured fields — read the specific attribute you need (the
  question, the answer, one SQL statement) rather than pulling and parsing whole content
  blobs.
- **Consent gating is a fact to report, not a failure.** Question/answer/SQL text may be
  blocked by the account's data-sampling settings — say so and continue from
  volume/failure structure.
- **Schema questions are answered from the normalized vocabulary** — see the span-field
  reference below; do not run exploratory queries against the trace store to discover
  fields.
- **Fix language:** space instructions, curated/annotated tables and column
  descriptions, example SQL / benchmark questions, or resolving the source-table data
  incident. Never code changes, never model or token tuning.

## Cross-links

- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Alert-type playbooks: `agent-alert-*.md`

# Backend: Databricks Knowledge Assistant / Agent Bricks (`databricks_mlflow_ka`)

## What this backend is

A **no-code** Agent Bricks Knowledge Assistant — a RAG assistant over document sets,
configured entirely through the Databricks UI. Each trace is one Q&A turn: the root span
carries the user's question and the final answer, and retriever spans carry the document
chunks the answer was grounded on. Its behavior is driven by its configuration — the
assistant's instructions and its **knowledge sources** — so the investigation centers on
retrieval grounding and document quality, not on code or models.

**CRITICAL: model and token fields are empty by design here — a token-usage, cost, or
model-swap question has no answer on this backend. Never chase token anomalies.**

## Signal available here

- **Per-turn traces** — each trace is one turn: a root span with the question and final
  answer, chain steps, and retriever tool-call spans. (`get_agent_trace` does not read
  this backend — it errors; span-grain detail comes from `run_troubleshooting_agent`,
  and manual reads work at turn grain via `get_agent_traces`.)
- **Retrieval grounding (the differentiator)** — the retriever spans' tool-call output
  holds the retrieved document chunks. What was asked, what was retrieved, and whether
  the answer was grounded in it is THE signal on this backend. Chunk content is
  consent-gated; retrieval counts and structure are always readable.
- **Failure status per span** — failed turns carry a Knowledge-Assistant-specific error
  marker; error rates and failed-turn trends are readable via `get_agent_traces`.
- **Volume and latency trends** — turns, retrievals, failed turns, duration, via
  `get_agent_traces` aggregation.
- **A config surface** — instructions plus the knowledge sources (the document sets it
  retrieves from), the platform analog of code history.

## Absent by design — do not chase

- **No model, no tokens** — always null. No cost or model-swap framing applies.
- **No agent source code and no PRs** — the assistant is configured, not coded.
- **Conversation grouping is usually absent** — `conversation_id` is frequently null;
  each trace is then a standalone turn. Do not rely on conversation reads
  (`get_agent_conversations` / `get_agent_conversation` may come back empty — that is
  expected, not a data problem; work at turn grain with `get_agent_traces` instead).
- **Eval scores are not in the spans** — quality scores are Monte Carlo–computed and
  live on the monitor/alert.
- **No SQL and no lineage** — a Knowledge Assistant generates no SQL, so there are no
  source tables to extract from queries; retrieval grounding is the analogous
  root-cause bridge.

## Investigation approach

1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`.
2. **Inspect the retrieval grounding FIRST (the primary root-cause move).** For the
   failing or low-scored turns, look at what the retriever spans fetched. Missing,
   stale, or off-topic chunks point at the knowledge source — a document set that
   changed, went stale, or lost coverage — not at the assistant itself. This is the KA
   analogue of tracing a wrong answer back to its data source.
3. **Compare BEFORE vs AFTER.** Incident window vs the prior baseline: turns, traces,
   retrievals, failed turns, duration — **not tokens**. A step on a specific day points
   at a knowledge-source or config change that day.
4. **Look at WHAT happened — and rule out a false positive.** Read the failing turns
   (and, when consent allows, the actual question/answer pairs). Genuine problem (bad
   grounding, real error spike) vs false positive (legitimately hard questions, an
   expected spike, a too-tight threshold). A benign breach is a finding — say so and
   recommend adjusting the monitor.
5. **Diff the config surface against the onset** — the instructions and the list of
   knowledge sources. Check `get_alerts` for data incidents on the tables/documents
   behind the knowledge sources.
6. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
   collect results with `get_troubleshooting_agent_results`.

## Gotchas

- **Never produce PR, token, cost, or model findings.** Evidence must be about the
  retrieval grounding, the failure/volume trend, or the assistant's configuration.
- **Consent gating is a fact to report, not a failure.** Question/answer text and
  retrieved chunks may be blocked by the account's data-sampling settings — say so and
  continue from retrieval counts and failure structure.
- **Don't lean on conversation ids** — treat each trace as a standalone turn unless the
  data proves otherwise.
- **Schema questions are answered from the normalized vocabulary** — see the span-field
  reference below; never run exploratory queries against the trace store to discover
  fields.
- **Fix language:** adjust the assistant's instructions, refresh or fix a knowledge
  source, or resolve the data incident behind it. Verification steps are grounding
  checks ("inspect the retrieved chunks for the failing turns", "diff the knowledge
  sources against the onset date") — never "review the PR" or model/token tuning.

## Cross-links

- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Alert-type playbooks: `agent-alert-*.md`

# Backend: Databricks MLflow SDK / Agent Bricks (`databricks_mlflow_sdk`)

## What this backend is

A **customer-coded** Databricks agent built with the Mosaic AI Agent Framework (Agent
Bricks SDK). MLflow autologging captures a real OTel-shaped span tree — with model and
token data — into Unity Catalog tables that Monte Carlo reads through a normalized view.
Of all the Databricks-family backends this is the closest to the managed OTel store in
investigation shape: it is a code agent, and model/token/PR findings are all valid.

**CRITICAL: this agent is identified by its Databricks coordinates
(database/schema/agent name), not by a resolvable trace-table name. Take the agent
reference from `get_agent_metadata` verbatim — do not try to locate or name a trace
table yourself.**

## Signal available here

- **Real multi-span trace tree** with parent/child structure, per-span timing, and
  status — captured in the normalized data. There is no direct MCP span-tree read here
  (`get_agent_trace` is managed-store-only and errors on this backend): span-grain
  drill-down goes through `run_troubleshooting_agent`; manual reads work at trace
  grain via `get_agent_traces`.
- **Model and token counts per span** — model-swap, cost, and context-growth questions
  have answers here (unlike Genie and the Knowledge Assistant).
- **Workflow / task segmentation** — customer-set attributes broadcast trace-wide;
  enumerate values with `get_agent_segments`, aggregate with `get_agent_traces`.
- **Error status and error detail** per span.
- **Change correlation** — a code agent: when the customer has GitHub connected,
  correlate the onset with merged PRs and deploys.

## Absent by design — do not chase

- **No free-form attribute exploration** — attributes were flattened into the standard
  fields at normalization; the standard vocabulary is everything there is. Don't dig
  for extra attribute keys.
- **Eval scores are not in the spans** — quality scores are Monte Carlo–computed and
  live on the monitor/alert, not in trace data.
- **Conversation clustering and conversation-grain eval breaches** exist only on the
  Monte Carlo–managed OTel store and the Cortex/Genie platform backends.
- **Raw content is consent-gated** — without the account's data-sampling consent,
  reason from span taxonomy, status, token distribution, and per-node latency.

## Investigation approach

1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`
   (note the coordinate-style agent reference).
2. **Follow the managed-OTel playbook** (see `agent-backend-clickhouse.md`): establish
   latency / error-rate / throughput / token trends over ~7 days before the onset to
   1 day after via `get_agent_traces` aggregation; find the onset; decide step vs drift.
   There is no built-in previous-period baseline — build your own comparison window.
3. **Segment the regression** — break the moved metric down by workflow / task / model
   (`get_agent_segments`); a regression confined to one node or one model is a
   different root cause than a fleet-wide one.
4. **Classify errors before hypothesizing** — provider rejection, timeout, fast-fail,
   parse failure, code exception, slow-but-healthy — and get a known-bad trace's
   failed-spans-in-order view from the automated run (`run_troubleshooting_agent`;
   earliest/innermost failure first). `get_agent_trace` errors on this backend —
   manual MCP reads stop at trace grain.
5. **Compare content across cohorts** when consent allows — breaching vs pre-onset
   prompts and completions.
6. **Correlate with changes** — PRs merged before the onset, provider status pages,
   model changelogs. Model-switch and context-overflow plays from the managed-OTel
   playbook apply in full.
7. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
   collect results with `get_troubleshooting_agent_results`.

## Gotchas

- **LLM spans are marked by request type, not by span-name conventions.** On this
  backend an LLM call is identified as a "chat"-type span — do not pattern-match span
  names (the `.chat` suffix taxonomy belongs to the managed OTel store).
- **Schema questions are answered from the normalized vocabulary** — see the span-field
  reference below; never run exploratory queries against the underlying tables to
  discover fields.
- **Consent gating is a fact to report, not a failure** — if content reads are blocked
  by the account's data-sampling settings, say so and continue from structure and
  tokens.
- **PR, token, and model evidence are all valid here** — this is the Databricks backend
  where those questions DO have answers; don't import Genie/Knowledge-Assistant
  restrictions.
- Managed-OTel gotchas carry over: interrupt-style control-flow messages are not
  errors, exception text quoted in input context is not a real error, and a PR merged
  after onset cannot be the root cause.

## Cross-links

- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Managed-store playbook: `agent-backend-clickhouse.md`
- Alert-type playbooks: `agent-alert-*.md`

# Direct Trace Intake: No Alert

Use this when the user brings a `trace_id`, `span_id`, or `conversation_id` — or just a
plain description ("this trace failed", "the bot gave a wrong answer yesterday") —
without a Monte Carlo alert.

## Goal

Resolve **which agent**, **which backend**, and **whether an alert already covers
this** — and only then investigate the specific traces.

> **CRITICAL:** `run_troubleshooting_agent` requires a Monte Carlo alert/incident UUID.
> Without an alert, this path is manual-only — never pass a trace or conversation ID to
> it.

## Steps

### 1. Resolve the agent

Call `get_agent_metadata` and match the user's agent by name or reference. If more than
one agent plausibly matches, **ask the user which one** — do not pick.

> **NEVER** guess the backend from an agent's name. On this path the backend comes from
> the agent's `backend_class` in the `get_agent_metadata` response — the same server-side
> classification the alert path gets from `get_alert_agent_classification` (which is
> alert-scoped and cannot be used here). If `backend_class` is null (an agent the server
> could not classify, or an older Monte Carlo server), ask the user which backend applies
> rather than assuming.

### 2. Search for a matching agent alert

Call `get_alerts` over the relevant window (last 7–14 days, or around the trace's
timestamp), looking at the agent alert categories: `"Agent evaluation"`,
`"Agent metric"`, `"Agent trajectory"`, `"Agent validation"`. Match on the same agent,
timeframe, and symptom (a failing trace often sits inside a metric or validation
breach; a wrong answer often sits inside an evaluation breach).

**If a matching alert exists, treat the user as having provided that alert** and
re-enter the main `SKILL.md` flow at Step 1 — Step 1.5 there kicks off
`run_troubleshooting_agent`, and `get_alert_agent_classification` gives you the shape
and backend. The alert path gets you the resolved breaching set and automated
troubleshooting for free.

### 3. Investigate the supplied items directly

No matching alert — manual investigation, strictly scoped to what the user brought:

- **`trace_id`** → `get_agent_trace`: read the span tree. Failed spans first — in a
  cascade of failures the root cause is usually the earliest or innermost failing span.
  Then timing (where the duration goes), token counts per span, and the model on each
  LLM span. (Managed-store (`ao_clickhouse_otel`) agents only — on other backends
  `get_agent_trace` errors; work from `get_agent_traces` — per-trace status, error
  counts, tokens, duration — and the conversation reads. This path has no automated
  run to lean on.)
- **`conversation_id`** → `get_agent_conversation`: read the thread turn by turn, find
  the turn where things went wrong, then `get_agent_trace` on that turn's trace for the
  span-level view (managed-store agents only — see above).
- **Description only** → `get_agent_traces` filtered by the described symptom (errors,
  latency, the time window the user gives) to find candidate traces first, then drill
  in as above.

Raw content (prompts, completions, transcripts) is gated on the account's data-sampling
consent; without it, reason from structure (span taxonomy, status, tokens, durations)
and say so.

### 4. Widen to the cohort

A single trace is only interpretable against its population. Pull the surrounding
window — roughly 7 days before the trace to 1 day after — with `get_agent_traces` for
the same agent and workflow, and use `get_agent_segments` for workflow/task/model
breakdowns. Answer: is this failure unique, or part of a trend? If a trend, when did it
start, and what changed at the onset (prompt, model, config, deploy)?

### 5. Match depth to the question

- **Lookup questions** ("what model was used?", "list the spans in this trace") →
  answer directly from the trace read; no root-cause pipeline.
- **"Why" questions** ("why did this fail?", "what changed?") → the full treatment:
  cohort comparison, onset dating, change correlation.
- When in doubt, prefer the rigorous path — more rigor is always safe; a raw data dump
  in place of an answer is not.

## Reading the results

- **Stay scoped.** The user asked to troubleshoot *these* items — do not invent an
  incident or a breach framing around them.
- A trace that is anomalous against its cohort (only trace failing, 10× the usual
  tokens) points at something specific to its inputs; a trace that matches a degraded
  cohort points at a population-level change — investigate the onset, not the single
  trace.
- If the widened view reveals a population-level problem with no monitor watching it,
  suggest creating one (the monitoring-advisor skill) — next time there will be an
  alert, and the troubleshooting agent can run automatically.

## Common mistakes

| Mistake | Why it fails / what to do instead |
|---|---|
| Passing a trace/conversation ID to `run_troubleshooting_agent` | It requires an alert/incident UUID; this path is manual-only |
| Guessing the backend from the agent's name | Read `backend_class` from `get_agent_metadata` — never name heuristics |
| Skipping the `get_alerts` check | A matching alert gives you the resolved breaching set, classification, and automated troubleshooting |
| Judging a single trace in isolation | Always compare against its cohort before concluding |
| Running a full root-cause pipeline for a lookup | Match depth to the question |
| Inventing an incident framing | Report on the supplied traces; widen for context, not for drama |

## Related references

- Backend-specific signal and gotchas: the `agent-backend-*.md` file for the backend you
  resolved (the same files the router selects on the alert path).