{
  "schema_version": 1,
  "skill_name": "agent-evals-and-observability",
  "evals": [
    {
      "id": "eval-dataset-design",
      "prompt": "I want to build an evaluation set for our customer-support agent to judge whether responses are good before every release. What does the dataset design look like: what cases belong in it, how many, and how do I keep it from going stale?",
      "expected_output": "An eval dataset design organized around the agent's task contract: the response defines a set of representative tasks sampled from real production traffic (the actual questions and edge cases users ask), each with a task description, the input, and the expected behavior, and it explains the population discipline: cases are drawn from production distributions including the failure modes the team cares about, plus a curated set of edge cases and regression cases from past incidents. The dataset's size is matched to the decision it supports (a small curated set for release gating versus a larger sampling set for tuning), and the response covers the freshness contract: a process for adding new cases from production incidents and removing or re-validating cases that no longer reflect the product, with versioning so results stay comparable across releases.",
      "assertions": [
        "The dataset is sampled from real production traffic and past failure modes, not invented by the team alone",
        "Each case carries a task description, input, and expected behavior",
        "Dataset size is matched to the decision it supports",
        "A freshness process adds incident-derived cases and retires stale ones",
        "Versioning keeps results comparable across releases"
      ]
    },
    {
      "id": "grader-criteria",
      "prompt": "Our eval harness has tasks and recorded agent outputs, but the scoring is a single human judgment of 'looks good.' I want objective, repeatable grading. How do I write grader criteria that multiple reviewers (or a judge model) can apply consistently?",
      "expected_output": "Grading criteria written as discrete, checkable requirements rather than holistic impressions: each task has pass-fail or rubric-scored criteria derived from the task contract \u2014 required content must be present, required steps must be taken, forbidden behaviors must be absent, and correctness is defined against a reference answer or verifiable facts rather than style. The response explains the rubric design: a small number of criteria (not a long checklist of trivia), each stated so that a reviewer can determine pass or fail without interpretation, with explicit handling of partial credit and a rule for when a single failure fails the whole case (e.g., a safety violation or a wrong factual claim). It also covers calibration: sample-scoring a set of outputs against the rubric, reconciling disagreements, and iterating the rubric until reviewers converge.",
      "assertions": [
        "Criteria are discrete and checkable, derived from the task contract",
        "Required content, required steps, and forbidden behaviors are separated",
        "Partial credit and fail-the-case rules are explicit",
        "A calibration pass with disagreement reconciliation is prescribed",
        "Rubric iteration continues until reviewers converge"
      ]
    },
    {
      "id": "regression-analysis",
      "prompt": "Our agent's overall score went up after a prompt change, but a few individual tasks got much worse, and I suspect they are the ones that matter. How do I analyze eval results across releases to catch regressions rather than a single averaged number?",
      "expected_output": "A regression analysis that looks below the aggregate score: the response prescribes comparing per-case results between the baseline and candidate release, separating the cases that improved, regressed, and stayed the same, and weighting the regressed cases by their production frequency and severity so a drop on a high-traffic task outweighs gains on rare ones. It explains the statistical ground rules: small eval sets produce noisy deltas, so the analysis distinguishes meaningful changes from sampling noise (via confidence bounds or a stated sample requirement) and flags regressions for investigation even when the average improves. It also covers bucketing by case category (customer-visible errors, safety, style) so the team can see which behavior class moved, and it prescribes a fix loop: investigate the regressed cases, decide whether the change or the eval is wrong, and re-run.",
      "assertions": [
        "Analysis compares per-case results between baseline and candidate, not just the average",
        "Regressed cases are weighted by production frequency and severity",
        "Statistical noise in small eval sets is accounted for with confidence bounds",
        "Results are bucketed by behavior class such as safety, errors, and style",
        "A fix loop investigates regressed cases and re-runs after changes"
      ]
    },
    {
      "id": "release-gate-design",
      "prompt": "We want to gate releases on eval results so a bad change cannot ship. How do I design the release gate so it blocks real regressions without making every release an evals fire drill?",
      "expected_output": "A release-gate design that separates the gating decision from the raw score: the response defines the gate criteria in terms of the regression analysis rather than a single threshold \u2014 no regressions on critical-path cases, no regressions beyond a tolerance band on the overall set, and the gate uses the previously established baseline for the same version of the eval set (so set changes do not silently move the goalposts). It prescribes the operational mechanics: the gate runs in CI, produces a comparable report against the merged baseline, blocks on the blocking criteria, and routes borderline results to a human review queue with the evidence attached rather than an automatic pass or fail. It also covers the escape hatch: an explicit override process with a recorded reason and owner, so the gate stays credible.",
      "assertions": [
        "The gate uses regression-relative criteria, not a single absolute score",
        "The baseline is pinned to the same eval-set version so goalposts do not move",
        "The gate runs in CI with a comparable report and blocks on critical-path regressions",
        "Borderline results route to human review with evidence, not automatic pass or fail",
        "An explicit, recorded override process keeps the gate credible"
      ]
    },
    {
      "id": "incident-to-case-learning",
      "prompt": "A customer-facing incident last week traced back to an agent answer we never tested: the agent confidently gave wrong configuration advice. I want the incident to become a permanent eval case so it cannot regress. What is the incident-to-case workflow and how does trajectory review fit in?",
      "expected_output": "An incident-to-case workflow that converts the postmortem into durable eval coverage: the response walks the process \u2014 extract the failing behavior from the incident (the wrong output, the context that produced it, the harm), turn it into a task case with the correct expected behavior and grader criteria, add it to the regression set, and verify it fails on the current release and passes on the fix. It explains where trajectory review fits: for agent failures, the answer alone may not show the flaw, so the review examines the reasoning trajectory (the steps, tool calls, and sources the agent used) to understand whether the error was a knowledge gap, a retrieval failure, or a reasoning failure, which determines the fix and the right case shape. The response also covers the loop: the new case joins the baseline, so any future release that reintroduces the behavior is blocked.",
      "assertions": [
        "The workflow converts the incident into a task case with expected behavior and grader criteria",
        "The case is verified to fail on the current release and pass on the fix",
        "Trajectory review is used to classify the failure as knowledge, retrieval, or reasoning",
        "The failure classification drives both the fix and the case shape",
        "The new case joins the baseline so the regression is blocked going forward"
      ]
    },
    {
      "id": "diagnostic-performance-replay",
      "prompt": "A candidate agent scores higher on the frozen answer set, but its replayed tool calls time out under production concurrency and one rare customer slice regresses. Design the evidence and release interpretation.",
      "expected_output": "Separate performance, diagnostic, replay, and online evidence. Compare paired case and slice outcomes, inspect timeout and trajectory causes, validate replay fidelity and side-effect suppression, and hold or condition release because the rare slice and latency failure are not erased by the aggregate score.",
      "assertions": [
        "Separates performance, diagnostic, replay, and online evidence",
        "Reports per-case and slice results rather than only an aggregate",
        "Checks replay fidelity and side-effect suppression",
        "Treats timeout and rare-slice regression as release-relevant evidence"
      ]
    },
    {
      "id": "offline-online-metric-mirror",
      "prompt": "Our offline grounded-answer score improved, while online thumbs-up rate rose only for heavy users and fell for new users. Create a metric mirror and decide whether this proves a quality improvement.",
      "expected_output": "Record the offline property, online denominator and population, assumed relationship, slices, missingness and selection effects, feedback-loop risks, and validation windows. Treat the result as exploratory or inconclusive until the relationship is checked across comparable windows; do not claim proof from the top-line thumbs-up rate.",
      "assertions": [
        "Names the offline and online populations and denominators",
        "Inspects slice divergence and feedback or selection bias",
        "Records the assumed link and validation windows",
        "Does not treat the online top-line signal as proof"
      ]
    },
    {
      "id": "judge-abstention-stability",
      "prompt": "A pairwise model judge prefers candidate A 80% of the time, but reversing answer order changes that to 52%, and repeated samples disagree on long answers. What should the evaluation system do?",
      "expected_output": "Freeze the rubric and judge configuration, inspect order and length effects on held-out calibration cases, report disagreement and variance, and abstain or escalate these comparisons rather than gating on the unstable score. Revalidate after judge or rubric changes.",
      "assertions": [
        "Tests reversed order and length-related disagreement",
        "Reports variance and calibration evidence",
        "Uses abstention or escalation for unstable comparisons",
        "Does not gate on the 80% score alone"
      ]
    },
    {
      "id": "grounding-retrieval-generation-attribution",
      "prompt": "A RAG assistant gives a fluent answer with an unsupported claim. The retriever returned no relevant authorized document for that claim. How should the case be graded and routed?",
      "expected_output": "Decompose the response into claims, record the evidence and authorization status, classify this claim as a retrieval or authorization/filtering failure, and grade factual support separately from fluency. Add a regression case and do not ask a judge to infer support without the source context.",
      "assertions": [
        "Uses claim-level grading with evidence and authorization status",
        "Attributes absent relevant evidence to retrieval or filtering",
        "Separates factual support from fluency",
        "Adds durable regression coverage"
      ]
    },
    {
      "id": "grounding-generation-mixed-failure",
      "prompt": "The retriever returns an authorized, directly relevant policy paragraph, but the final answer contradicts it. Explain the diagnosis and required evidence.",
      "expected_output": "Verify the retrieved source and its authorization, preserve the relevant evidence span, classify the primary failure as generation or grounding, retain an interaction label if retrieval context was malformed, and test the corrected behavior with a claim-level regression case.",
      "assertions": [
        "Verifies source relevance and authorization",
        "Attributes contradiction with present evidence to generation or grounding",
        "Allows multiple labels when component failures interact",
        "Creates a claim-level regression case"
      ]
    },
    {
      "id": "counterfactual-missing-support",
      "prompt": "Our agent always selected retriever A in last month\u2019s logs. We have answer confidence and observed satisfaction, but no action-selection probabilities. A new policy selects retriever B for half the requests. Estimate the uplift from these logs and approve rollout if positive.",
      "expected_output": "Decline an identifiable uplift/rollout claim from unsupported logs; distinguish confidence from propensity and specify the required evidence.",
      "assertions": [
        "Identifies missing action support for retriever B and does not fabricate its rewards",
        "Distinguishes answer confidence from action-selection probability",
        "Does not claim that a large log or doubly robust estimator repairs absent support",
        "Proposes bounded additional evidence and routes statistical estimation to its owner without authorizing rollout"
      ]
    }
  ]
}

# Datasets and Fixtures

Freeze each evaluated dataset version as an immutable artifact under the repository's chosen naming scheme. Record a changelog rather than prescribing a particular version-label convention. Preserve the exact case, fixture, configuration boundary, and manifest needed to reproduce a run.

The manifest must state provenance; rights and consent; creation and review owners; task taxonomy; slices; source class; expected outcomes; fixtures and reset method; expected side effects; contamination and leakage risk; limitations; access; retention; deletion handling; and change history.

Use isolated sandboxes or resettable fixtures for stateful tools. Inspect actual environment state to grade idempotency or side effects; identical text is not sufficient evidence. Treat production-derived data as a controlled source: verify rights, minimize and transform it, redact before storage, limit access, define deletion, and prevent it from entering prompts or training/evaluation targets where it would contaminate comparisons.

Report prevalence-oriented samples separately from deliberately risk-enriched or adversarial cases. Slices should expose meaningful differences such as task class, tool availability, locale, policy path, or input condition; they are not a license to infer outcomes for unrepresented groups.

# Evaluation Design

Write a task contract: inputs and context, permitted tools and permissions, expected outcome, prohibited outcomes, environment assumptions, and observable completion evidence. Add a trajectory contract when the path matters: eligible tool selection and arguments, authorization checks, state transitions, grounding sources, recovery behavior, stopping/escalation, and permitted side effects.

Select evidence rather than requiring every evaluation type:

| Evidence | Useful when | Boundary |
|---|---|---|
| Unit | A prompt formatter, tool adapter, parser, or guard can be isolated | Does not establish integration behavior |
| Component | Router, retriever, policy layer, or tool boundary interacts internally | May hide production environment behavior |
| Scenario | A complete task must run under controlled conditions | Results depend on fixture fidelity |
| Regression | A known behavior or escaped failure must remain protected | Cannot cover unknown failures |
| Adversarial | Misuse, injection, unsafe actions, or leakage are plausible | Challenge coverage is never exhaustive |
| Online | Distribution shift, service variability, or user outcomes matter | Confounding and consent constrain interpretation |
| Human review | Domain or subjective criteria cannot be directly automated | Review remains rubric- and context-sensitive |

Use synthetic, curated, replayed-production, adversarial, and regression cases as distinct declared sources. Choose a mix based on task harm, reversibility, novelty, external side effects, and availability of valid evidence.

# Evidence modes and offline-to-online metric mirrors

Choose the evidence mode from the decision, rather than treating offline scores as a complete proxy for production behavior.

| Mode | Question | Design | Boundary |
|---|---|---|---|
| Performance | Does the candidate meet the declared outcome on known cases? | Frozen cases, baseline pairing, task and slice metrics | Does not explain why a case fails or predict distribution shift |
| Diagnostic | Which behavior, slice, component, or condition produces the failure? | Error taxonomy, controlled perturbations, slice analysis, trace review | Diagnostic findings are hypotheses until the claimed property is tested |
| Replay/simulation | What would the candidate do under recorded or modeled conditions? | Versioned replay inputs, simulated tools/state, side-effect suppression, fidelity checks | Replay cannot establish user response or an alternate outcome without assumptions |
| Online | What happens under live distribution, latency, feedback, and user interaction? | Shadow, canary, or randomized exposure with guardrails and consent | Confounding, feedback loops, missingness, and proxy gaming constrain inference |

## Metric mirror

For every offline metric used to justify an online decision, record:

- offline property, dataset population, case version, and grader;
- online signal intended to reflect it, population and denominator;
- causal or operational link assumed between the two;
- expected direction and acceptable divergence;
- slice coverage and known blind spots;
- data latency, missingness, selection, and feedback-loop risks;
- owner, alert or review action, and next validation date.

Do not call two measures correlated because they share a name or trend in one release. Check the relationship over multiple comparable runs or windows, inspect reversals by slice, and record when evidence is insufficient. A product metric can be useful for monitoring while remaining unsuitable as a release oracle.

## Readiness and replay controls

Before live exposure, verify the candidate and baseline identifiers, fixture and tool versions, replay fidelity, side-effect suppression, time and concurrency conditions, and privacy limits. Shadow or replay evidence can qualify a candidate for a live experiment; it cannot silently authorize mutations. For replayed production data, retain only the minimum transformed inputs required for the decision and document rights, consent, retention, and contamination controls.

## Counterfactual readiness for discrete decisions

Use off-policy evaluation only when the logged decision and outcome can support the
question. For an agent, start with a bounded choice such as routing to one of several
retrievers or tools. An arbitrary generated response or full multi-step trajectory is
not automatically a contextual-bandit action with comparable logged outcomes.

Before handing estimation to `data-scientist`, record:

1. Decision unit, eligible actions, available context, chosen action, logging policy
   version and the probability with which that policy chose the action. Confidence in
   an answer is not its action-selection probability.
2. Outcome definition, observation horizon, missing/delayed outcomes and attribution.
   Log only necessary, permitted context; do not collect sensitive fields speculatively.
3. Target policy and population, plus overlap: where the target chooses actions the
   logging policy never chose, the log cannot identify those outcomes without additional
   assumptions. Do not invent rewards for unchosen actions.
4. Weight concentration, effective information, uncertainty, and sensitivity to any
   clipping or model assumptions. Large logged volume does not fix absent support.
5. The smallest next evidence step if assumptions fail: improve authorized logging,
   narrow the estimand explicitly, or propose a bounded controlled experiment. Do not
   promote based on a counterfactual score whose eligibility checks failed.

An outcome model can extrapolate beyond observed support, but that is model-dependent
prediction, not an observed counterfactual. Doubly robust methods do not make missing
support or deployment distribution changes disappear. Preserve unsupported regions and
route estimator choice, confounding and interval design to the statistical owner.

Primary research checked 2026-09-14: [Wang et al., optimal and adaptive off-policy evaluation](https://arxiv.org/abs/1612.01205)
for contextual-bandit evaluation assumptions, and [Kallus et al., distributionally robust off-policy evaluation](https://proceedings.mlr.press/v162/kallus22a.html)
for sensitivity to shifts between logged and deployment environments.

# Framework Routing

Use this skill for methodology, evidence, data governance, comparison, and privacy regardless of framework.

| Need | Route |
|---|---|
| PydanticAI test models, Pydantic Evals, or framework traces | [pydanticai](../../pydanticai/SKILL.md), especially `references/testing-evals.md` |
| LangGraph state, routing, checkpointers, or LangGraph implementation | [langgraph](../../langgraph/SKILL.md), especially `references/evals.md` |
| A general defect discovered by an eval | [systematic-debugging](../../systematic-debugging/SKILL.md) |
| General completion evidence | [verification-methodology](../../verification-methodology/SKILL.md) |
| Preventive AI, authorization, or data-boundary controls | [secure-software-engineering](../../secure-software-engineering/SKILL.md) |

Do not require an SDK, hosted evaluator, or tracing backend. Translate framework events into the task and trajectory contracts before comparing implementations.

# Grader Families and Calibration

Compose graders by the property each can observe; no universal taxonomy or single grader is sufficient.

| Family | Direct evidence | Common boundary |
|---|---|---|
| Deterministic/programmatic | Schema, arithmetic, exact state-independent properties | A regex or keyword does not prove semantics, safety, or grounding |
| Execution/environment | Sandbox state, permissions, tool results, side effects | Requires faithful fixtures and reset evidence |
| Human-rubric | Contextual/domain judgment against an anchored rubric | Training, fatigue, incentives, and context affect results |
| Model judge | Structured assessment at scale | Prompt/model changes and bias require calibration |
| Pairwise/ranking | Relative preference or ordering | Randomize position and blind identity; full round robin is only one costly design |
| Domain/safety | Specialist or policy criteria | Must define scope, false positives, false negatives, and escalation |

For model judges and subjective rubrics, hold out calibration material from tuning. Blind candidate identity, randomize order where relevant, inspect rationales, compare errors and disagreement by slice, and use an agreement/error analysis appropriate to label type and decision. Predefine acceptable error for the risk; do not impose universal rater counts, agreement thresholds, or interpretations of kappa/correlation. Revalidate after grader prompt, model, rubric, task distribution, or policy changes.

Human review is valuable evidence, not definitive truth. Resolve disagreement through rubric refinement, adjudication rules, or escalation appropriate to the decision, and retain the disagreement rather than averaging it away.

## Build A Grader Contract

For every grader, record the property, inputs visible to it, output schema, direction/scale, abstention and error behavior, reference material, versioned implementation or prompt, cost/latency constraints, slices where it is valid, known failure modes, and false-positive/false-negative consequences. A reference answer is evidence, not automatically correct; version and review it like any other oracle.

Deterministic checks should target properties such as parseability, exact identifiers, allowed tool names, numerical invariants, or observed sandbox state. They should not use keyword presence as a proxy for factual grounding or policy compliance. Execution graders should query the environment after the run, distinguish planned from committed effects, and verify cleanup or reconciliation.

A model judge needs the task contract, only the evidence required to judge it, an anchored rubric with counterexamples, an abstain/insufficient-context path, and a structured rationale. Do not ask a judge to verify external facts it cannot access. Blind candidate identity and irrelevant metadata; randomize order in pairwise work; test order reversal and legitimate format/length variation. Sampled pairs, tournaments, active selection, and full round robins are different designs with different coverage and cost.

## Calibrate And Revalidate

1. Freeze rubric, calibration cases, human-review instructions, and candidate identities before the comparison.
2. Collect independent labels where the decision requires them; preserve reviewer identity or role in restricted metadata so systematic disagreement can be investigated.
3. Compare confusion/error patterns and disagreement by task and slice. Select agreement statistics for the label scale and sampling design rather than treating one coefficient as universal.
4. Inspect rationales and adjudicate sampled disagreements. Distinguish rubric ambiguity, missing evidence, grader bias, reviewer error, and legitimately plural answers.
5. Define where the grader can gate, where it can screen for review, and where it must abstain. Freeze the accepted grader version.
6. Revalidate after changes to the grader model, prompt, rubric, reference answers, task distribution, policy, or telemetry available to the grader.

# Claim grounding and retrieval/generation attribution

Evaluate grounded systems at claim level when a response can contain several factual assertions. The goal is to identify which claims are supported by the supplied evidence and which component failed.

## Claim record

For each sampled response, record a minimized claim identifier, claim text only when approved by the data contract, evidence span or source identifier, support status (`supported`, `contradicted`, `not supported`, or `not verifiable`), and reviewer/grader version. Keep source access and sensitive content under the dataset's access and retention controls.

## Attribution sequence

1. Check whether the retrieval set contains sufficient, authorized evidence for the claim.
2. If evidence is absent or irrelevant, classify a retrieval miss or authorization/filtering failure.
3. If evidence is present but the answer contradicts or invents beyond it, classify a generation/grounding failure.
4. If both retrieval and generation are inadequate, retain both labels and avoid forcing a single root cause.
5. If the source itself is stale, conflicting, or unavailable, mark the case inconclusive and record the source limitation.

Retrieval precision/recall, answer relevance, faithfulness, and claim factuality measure different properties. Do not average them into one quality claim without a declared decision rule. A fluent or semantically similar answer is not evidence of factual support.

## Required cases

Include at least one supported claim, an unsupported fluent claim, a retrieval miss, an answer that ignores retrieved evidence, an unauthorized document that must not be used, and a stale/conflicting source. Grade source authorization and side effects separately from answer quality.

# Judge stability, abstention, and escalation

A model judge is an instrument with its own error profile. Validate it before using it as a gate.

## Stability protocol

Freeze the rubric, judge model and prompt, reference material, candidate identity masking, and sampling plan. On a calibration set held out from prompt tuning:

1. Compare judge labels with independent human or domain labels appropriate to the task.
2. Repeat stochastic judgments when variation could change the decision; report the distribution, not only the mean.
3. Randomize candidate order and rerun pairwise cases with the order reversed.
4. Inspect disagreement by task, slice, length, style, and evidence availability.
5. Test insufficient-context and adversarial/judge-injection cases.
6. Freeze the accepted judge version and revalidate after model, prompt, rubric, reference, policy, task distribution, or telemetry changes.

Position, verbosity, and self-preference effects are documented risks in published judge studies, but their magnitude is setting-dependent. Do not import a universal correction or agreement threshold.

## Structured judgment

Prefer atomic checklist items when a holistic score hides the cause of failure. Each item names the observable property, evidence visible to the judge, pass/fail or ordinal scale, counterexample, and consequence of an error. Keep rationales short and structured; they are diagnostic evidence, not private reasoning or proof.

Use a panel or stronger judge only when the decision risk and expected disagreement justify its cost. Predefine aggregation, disagreement handling, and the boundary at which the judge must abstain. An abstention is a valid result: route it to human or domain review, or hold the release when no safe fallback exists. Never convert missing evidence into a confident score.

# Metrics and Statistical Reasoning

Report a profile by task and slice: task outcome; grounding to supplied or retrieved evidence; tool selection, arguments, ordering, authorization, and side effects; safety; reliability and recovery; cost/resource use; and latency. Include failures, timeouts, abstentions, and missing data rather than conditioning only on completed runs.

Compare candidate and baseline on the same cases when feasible. Preserve paired outcomes, effect sizes, uncertainty intervals suited to the metric, and run-level conditions. For stochastic systems, repeat runs when stochastic variation can change the decision; report variability rather than treating one draw as stable.

Choose sample size through the decision risk, baseline rate, metric distribution, planned pairing, slice coverage, minimum detectable effect, and available resources. Do not use fixed sample quotas. Plan power or sensitivity before a consequential comparison. A non-significant result does not show equivalence.

Predeclare primary comparisons where practical. Investigate multiple comparisons, changing composition, missingness, base rates, selection effects, temporal/environment confounders, and aggregate reversals. Use correction or hierarchical interpretation appropriate to the question, and show slice results instead of relying on a blended average.

## Define Each Measure

A metric contract names the property; unit or label set; population and denominator; included failures, retries, and abstentions; aggregation and tail/slice views; preferred direction; data source; measurement error; baseline; decision threshold or comparison rule; owner; and failure behavior when data is missing. Token counts are not automatically cost, latency averages can hide tail harm, and user feedback is a selected noisy signal rather than ground truth.

Keep hard invariants categorical. Report ordinary indicators separately, for example:

- task outcome and completion evidence;
- grounded claims or evidence use, with the actual source context available to the grader;
- tool selection, arguments, authorization checks, ordering, and committed side effects;
- safety/privacy/policy outcomes and grader false-positive/false-negative behavior;
- reliability, retries, recovery, loops, timeouts, and escalation;
- latency distribution and time to useful outcome; and
- resource use and monetary cost from the billing/usage source that actually defines them.

## Choose Comparison Methods From The Data

For paired binary outcomes, preserve the discordant case pairs and use an interval/test suitable for paired proportions. For continuous or heavy-tailed outcomes, consider paired bootstrap, permutation, or a justified model and report distributional views. For ordinal or rubric scores, preserve the scale and rater structure instead of pretending equal numeric distance. If a release needs to establish practical equivalence or non-inferiority, predefine the acceptable margin and design for that question; failure to detect a difference is not equivalence.

Choose repetitions and sample size from estimated variance/base rate, minimum effect worth detecting, pairing, slice coverage, desired error risk, and decision consequence. Run sensitivity analysis when inputs are uncertain. If the design cannot resolve the decision, report insufficient evidence rather than converting a wide interval into a pass.

# Optional OpenTelemetry GenAI Interoperability

OpenTelemetry GenAI semantic conventions are an optional interoperability layer, not this skill's canonical data model or a prerequisite for evaluation. The `open-telemetry/semantic-conventions-genai` repository was checked at commit `63f8200eee093730ce845d26ce2aafb621b0807e` on 2026-07-13 and labeled Development. Verify current status and canonical names before adopting any convention.

If useful, map local concepts such as agent invocation, model request, tool execution, workflow step, evaluator result, duration, token/resource use, and correlation to the then-current conventions. Keep an internal event contract that works without OTel and avoid copied example hostnames, paths, prompts, model names, or histogram bucket boundaries.

Prompt and output content are opt-in and sensitive under this guidance. Do not enable content capture by default; apply the minimization, redaction, access, retention, deletion, and incident controls in [production observability](production-observability.md) first.

# Production Observability and Privacy

Observability provides diagnostic context, not an automatic evaluation verdict. Correlate minimized traces, structured logs, metrics, deployment/configuration identity, and user feedback with a non-identifying invocation identifier. Record sampling rules and bias: errors, high-cost paths, and opted-out traffic may be under- or over-represented.

Default to metadata such as operation class, outcome category, duration, resource counters, and redacted error class. Treat prompt/output text, tool arguments/results, retrieved content, intermediate state, credentials, and personal data as sensitive opt-in data. Minimize before export; use structured allowlists and redaction at the earliest boundary; test redaction; enforce least-privilege access; set purpose-bound retention; support deletion; and maintain incident response and access-audit procedures.

Sampling, retention, access, and alert conditions are local risk decisions. Record their rationale, limitations, failure behavior, and how a privacy incident is contained without expanding collection indiscriminately.

## Telemetry Contract

Define the question before the signal. For each trace, log, metric, or feedback field, record purpose, producer, schema/version, sensitivity, cardinality, sampling, redaction/minimization, access, retention/deletion, correlation, owner, and what happens if collection or export fails. Keep high-cardinality identifiers out of metric labels; use restricted traces or logs for case-level diagnosis.

Useful backend-neutral signals can include:

- invocation/workflow/step outcome and bounded duration;
- model/tool/policy/configuration version identifiers;
- error and stop categories, retry/loop/escalation counts, and tool outcome class;
- resource or billed-usage counters from their authoritative source;
- committed/rolled-back side-effect category; and
- evaluator name/version/result linked by an opaque non-identifying run ID.

Do not synthesize user, conversation, or tenant identity from content. Do not expose chain-of-thought as telemetry. If content is genuinely required for a bounded investigation, use an approved opt-in path with purpose limitation, field allowlists, transformation/redaction before export, restricted access, deletion, and auditable shutdown.

## Alerts, Feedback, And Incident Learning

Alerts should point to an actionable operational condition and owner; they do not establish why quality changed. Define the population, sampling path, expected delay, missing-data behavior, and false-positive/false-negative costs. User feedback, support tickets, judge scores, and anomaly detectors are candidate signals with selection and measurement bias, not automatic labels.

For an incident: contain harm; preserve the smallest authorized evidence; record deployment/configuration and sampling gaps; reproduce in a safe fixture; distinguish agent, tool, data, policy, infrastructure, grader, and telemetry failures; correct the system; then add a transformed regression case only after rights/privacy/contamination review. Verify the fix offline and at the deployed boundary without broadening production capture merely to make diagnosis easier.

# Regression, Triage, and Release Gates

For each change, identify the baseline, frozen dataset, configuration differences, primary decision metrics, uncertainty method, slices, and hard invariants. Compare paired runs where possible. A regression investigation begins with reproduction, trace and fixture review, configuration and environment differences, then a narrow root-cause hypothesis. Do not tune against the same incident case without recording contamination.

Turn confirmed incidents and near misses into candidates for new cases only after privacy, rights, redaction, and retention review. Classify whether the failure was task contract, trajectory, tool/environment, grader, data, deployment, or observability failure.

Risk-tier gates should have an authority, evidence owner, and one of: approve, approve with recorded conditions, hold for insufficient evidence, or block. Define hard invariants separately from statistical indicators. Examples of hard invariants include unauthorized action, unapproved sensitive-data disclosure, or an unreconciled harmful side effect. They cannot be offset by quality, cost, or latency results. Every gate records rollback/containment, monitoring during rollout, and the next review trigger.

## Compare Without Moving The Goalposts

Freeze candidate and baseline identifiers, dataset version, grader versions, fixtures, environment, sampling/randomization plan, primary comparisons, slices, and exclusions before the decision run. Use the same cases and comparable external conditions where feasible. Record provider/model drift, cache state, concurrency, tool/data versions, and instrumentation changes that could explain a delta.

Report case-level paired outcomes, slice deltas, uncertainty/effect size, hard-invariant results, failures/timeouts/missing data, cost/resource and latency distributions, grader disagreement, and observed confounders. Separate exploratory findings from preregistered release evidence. Do not tune on the held-out decision set and then report the same run as independent confirmation.

## Build The Gate

A release matrix should state for each dimension:

| Dimension | Evidence | Decision rule | Hard or statistical | Owner | Failure/insufficient-data action |
|---|---|---|---|---|---|
| Task/trajectory outcome | Cases, state checks, trace review | Risk- and slice-specific | Statistical or categorical | | |
| Safety/privacy/authorization/side effects | Adversarial cases and environment evidence | Explicit invariant | Hard | | |
| Reliability/recovery | Failure injection and deployed evidence | Workload-specific | Statistical plus hard cases | | |
| Cost/resource/latency | Metered distributions | Budget/SLO-specific | Statistical | | |
| Operability | Correlation, alerts, rollback rehearsal | Evidence checklist | Categorical | | |

The owner chooses a verdict from the documented outcomes, not from an unweighted average. Conditional approval names the condition, monitor, expiration/review trigger, and authority to halt. Rollout evidence must be linked to the exact released configuration, and rollback must account for state and side effects rather than only code version.

# Source and Status Index

Sources inform decisions; they become binding only when adopted by applicable policy, contract, or regulation. Versions and statuses were checked 2026-07-13 unless otherwise stated. Recheck rolling documentation and Development-status conventions before consequential use.

| Source | Exact status/version checked | Primary URL | Decision use |
|---|---|---|---|
| NIST Artificial Intelligence Risk Management Framework | NIST AI 100-1, AI RMF 1.0, January 2023; NIST reports revision work underway | https://doi.org/10.6028/NIST.AI.100-1 | Voluntary risk and governance framing, not an eval threshold catalog. |
| NIST AI RMF Generative AI Profile | NIST AI 600-1, July 2024 | https://doi.org/10.6028/NIST.AI.600-1 | Voluntary generative-AI risk considerations, not a release formula. |
| OpenTelemetry GenAI semantic conventions | Development status at repository commit `63f8200eee093730ce845d26ce2aafb621b0807e` dated 2026-07-08 | https://github.com/open-telemetry/semantic-conventions-genai/tree/63f8200eee093730ce845d26ce2aafb621b0807e/docs/gen-ai | Optional moving interoperability guidance; prompt/output content is opt-in and sensitive. |
| PydanticAI testing documentation | Rolling official documentation checked 2026-07-13 | https://ai.pydantic.dev/testing/ | PydanticAI-specific unit and model-substitution implementation. |
| Pydantic Evals documentation | Rolling official documentation checked 2026-07-13 | https://ai.pydantic.dev/evals/ | Pydantic-specific dataset, experiment, evaluator, and span-evaluation APIs. |
| LangGraph testing documentation | Rolling official documentation checked 2026-07-13 | https://docs.langchain.com/oss/python/langgraph/test | LangGraph-specific node, graph, state, and partial-execution testing. |
| LangSmith evaluation concepts | Rolling vendor documentation checked 2026-07-13 | https://docs.langchain.com/langsmith/evaluation-concepts | Vendor implementation example only; its fixed example counts and product workflow are not methodology requirements. |
| “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” | Zheng et al., arXiv:2306.05685 v4, 2023-12-24 | https://arxiv.org/abs/2306.05685v4 | Evidence that judge behavior can include position, verbosity, and self-enhancement biases in the studied setting; not universal bias magnitudes. |
| “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” | Liu et al., arXiv:2303.16634, 2023 | https://arxiv.org/abs/2303.16634 | Example of rubric-guided model judging; use as research context, not a universal judge protocol. |
| “SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models” | Manakul et al., EMNLP 2023 | https://aclanthology.org/2023.emnlp-main.557/ | Consistency-based hallucination signal and its limits; not a factuality oracle. |
| “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation” | Min et al., EMNLP 2023 | https://aclanthology.org/2023.emnlp-main.741/ | Claim-level factual precision framing; source attribution and domain limits remain decision-specific. |
| Ragas documentation | Rolling documentation; verify API and metric definitions before use | https://docs.ragas.io/ | Named implementation example for RAG evaluation; metric names do not establish a release threshold. |

# Synthetic Scenario Probes

Use synthetic placeholders only. Do not put plausible credentials, identifiers, or personal data into fixtures.

| Probe | Contract and evidence |
|---|---|
| Multi-domain routing | A request needs two declared domains. Verify complete routing/coverage, permitted handoffs, and no unsupported extra domain. |
| Tool error and side effect | A sandboxed tool reports a transient error after a declared partial action. Verify state inspection, safe recovery or escalation, and no duplicate side effect. |
| Injection and privacy | Untrusted retrieved text requests disclosure or instruction override. Verify policy path, no sensitive content in telemetry, and redaction test evidence. |
| Grader calibration/disagreement | Blinded candidate outputs include legitimate variation and ambiguous cases. Compare rubric, judge, and reviewer disagreement; record resolution and limitations. |
| Regression detection | Candidate and baseline run the same frozen cases with recorded configuration. Report paired deltas, intervals, slices, missingness, and an insufficient-evidence outcome when warranted. |
| Privacy-safe incident diagnosis | A synthetic trace has only opaque correlation IDs, redacted fields, error category, and state summary. Verify diagnosis can identify the next evidence request without recovering raw content. |

Run each probe through its selected graders and release-gate template. A probe passes only when its stated evidence exists; it does not certify unrelated properties.

# Trace and Trajectory Review

Review trajectories when correctness, safety, cost, or recovery depends on how the agent acted. Use the trace-review template to inspect:

- tool selection, argument validity, ordering, and permission checks;
- state transitions, retries, recovery, stopping, escalation, and loop controls;
- grounding: retrieval, tool-result use, citations, and unsupported claims;
- planned and observed side effects, including compensating or rollback actions;
- correlation between final outcome, latency, resources, errors, and environment state.

Specify permitted alternatives rather than forcing a single path when more than one safe trajectory can satisfy the contract. Grade environment-visible effects for idempotency and recovery. Capture the smallest trace fields needed for the claim; redact before review and record inaccessible evidence as a gap.

## Minimal Run And Step Contract

Keep a backend-neutral internal record even when exporting to a tracing system:

- run/case/experiment identifiers and parent-child correlation;
- dataset, agent/workflow, prompt/policy, model/provider, tool, grader, code, fixture, and deployment versions;
- environment and permission boundary;
- step type, start/end or sequence, outcome/error category, retry/attempt relation, and stop reason;
- selected tool and bounded/redacted arguments/result summary;
- state transition and planned/committed/rolled-back side effects;
- grounding/evidence identifiers that can be resolved under access control; and
- final outcome plus linked grader results.

Do not manufacture a conversation ID from content, expose chain-of-thought, or copy raw inputs merely to make a trace look complete. Record when sampling or access removed evidence.

## Review Protocol

1. Reconstruct the intended task and trajectory contracts without looking at the final score.
2. Follow the state and permission boundary through each step; compare tool arguments and observed effects to the contract.
3. Locate the first divergence, not only the last visible error. Distinguish model choice, orchestration, tool, data, environment, policy, and telemetry failures.
4. Test whether recovery avoided duplicate or unreconciled effects and whether stopping/escalation happened at the right boundary.
5. Compare final-answer quality with path quality. A correct answer from an unauthorized or fragile path still fails the trajectory contract.
6. Record evidence gaps, alternative valid paths, root-cause hypotheses, and the case/fixture changes needed before rerun.

# Dataset Manifest

- Dataset name and immutable version identifier:
- Purpose, task taxonomy, and source class: synthetic / curated / replayed-production / adversarial / regression
- Provenance, rights/consent, creator, reviewer, and creation date:
- Cases, slices, expected outcomes, and known limitations:
- Fixture locations, environment version, reset method, and expected side effects:
- Contamination/leakage risks and controls:
- Access controls, retention, deletion path, and incident contact:
- Change history and superseded versions:

# Eval Plan

## Decision and Risk

- Change and decision:
- Owner and authority:
- Harm, reversibility, and deployment stage:
- Task contract and prohibited outcomes:
- Trajectory contract and permitted side effects:

## Evidence Design

| Evidence type | Risk addressed | Dataset/version and slices | Grader | Limitations |
|---|---|---|---|---|
| | | | | |

## Comparison and Analysis

- Baseline and candidate configuration:
- Pairing and stochastic-repeat plan:
- Primary metrics, effect sizes, uncertainty method, and minimum detectable effect rationale:
- Missingness, base-rate, multiple-comparison, and confounder handling:
- Privacy, rights, access, retention, deletion, and contamination controls:

## Decision Rule

- Hard invariants:
- Statistical indicators and insufficient-evidence disposition:
- Rollback/containment and monitoring:

# Grader Specification

- Property claimed and observable evidence:
- Grader family and why it fits:
- Inputs, output schema, rubric anchors, and abstain/escalate behavior:
- Known blind spots, false-positive/false-negative harms, and affected slices:
- Calibration set isolation and candidate blinding/order randomization:
- Human/model disagreement analysis and adjudication rule:
- Revalidation triggers: model, prompt, rubric, policy, task distribution, or environment change
- Version, owner, access, retention, and audit evidence:

# Offline-to-online metric mirror

- Decision, owner, and risk:
- Candidate/baseline and configuration versions:

| Offline property | Offline population/cases and grader | Online signal and denominator | Assumed link | Expected divergence | Slices/blind spots | Action/owner |
|---|---|---|---|---|---|---|
| | | | | | | |

- Sampling, missingness, consent, retention, and privacy controls:
- Feedback-loop, proxy-gaming, and confounding risks:
- Validation runs/windows and observed relationship:
- Evidence status: supported / exploratory / inconclusive:
- Next review trigger:

# Risk-Tiered Release Gate

- Change, risk tier, decision authority, and evidence owner:
- Dataset/run/trace report references:

| Category | Hard invariant or indicator | Evidence | Status | Owner/action |
|---|---|---|---|---|
| Safety/privacy/authorization | | | | |
| Task and trajectory outcome | | | | |
| Reliability/recovery | | | | |
| Cost/resources and latency | | | | |

- Decision: approve / approve with conditions / hold for insufficient evidence / block
- Rollback or containment path:
- Rollout monitoring, retention/deletion confirmation, and review trigger:

# Run Report

- Candidate, baseline, dataset version, fixture/environment, and run date:
- Pairing, stochastic repeats, failures/timeouts, and missingness:

| Dimension and slice | Baseline | Candidate | Effect and uncertainty | Interpretation |
|---|---|---|---|---|
| | | | | |

- Multiple-comparison and confounder handling:
- Trajectory and side-effect findings:
- Grader calibration/disagreement status:
- Privacy and provenance exceptions:
- Conclusion: improve / regress / inconclusive, with evidence gaps:

# Trace Review Record

- Invocation/correlation identifier (non-identifying):
- Task and trajectory contract version:
- Dataset case or approved production-sample provenance:
- Tool selection, arguments, ordering, permissions, and observed environment state:
- State transitions, grounding evidence, recovery, stopping, escalation, and side effects:
- Outcome, errors, latency/resources, and redaction/access limitations:
- Findings, evidence links, root-cause hypothesis, and follow-up owner: