Claude Skill

evals-analyze

Analyze evaluation results and close the loop. Specification failures create local CDRs to fix agent rules; generalization failures go to evaluator backlog.

LLM Mart · 0 points · 9 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download tikalk-adlc-team-skills-skills_evals_evals-analyze-3035db2.zip · 3 KB

tikalk/adlc-team-skills 137 1 forks MIT Updated 3d ago

Part of tikalk/adlc-team-skills — 21 skills

Install

skills CLI npx skills add https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-analyze

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tikalk-adlc-team-skills@llmmart

Git git clone https://github.com/tikalk/adlc-team-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tikalk/adlc-team-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

evals-analyze

What this skill does

Provides cross-functional team elevation and closed-loop feedback following EDD Principle VIII (Close the Production Loop) by deep-analyzing trajectory failure traces and routing them to correct resolution pathways.

Output:

Trajectory Analysis - Full multi-turn trace analysis with tool calls and context preservation (EDD Principle V)
Failure Routing:
- Specification Failures (agent logic missing/ambiguous) → subclassified (EVAL-010):
  - Mechanical (fixed, checkable pattern) → fix/extend the existing grader or add a unit test — a deterministic check, not a CDR
  - Judgment gap (missing intent, ambiguity, needs context) → automatically triggers a local call to team-learn to propose new context rules in adlc branch drafts/cdr/ to fix agent behavior
- Generalization Failures (grader flawed or lacks edge-case coverage) → Appends evaluator backlog items to the project backlog for ongoing monitoring.
Cross-Functional PR - Creates a team-ai-directives PR with insights and rule updates (EDD Principle X)

Key EDD Principles Applied:

Principle VIII: Close Production Loop - Spec failures → fix directives; Gen failures → evaluator backlog
Principle V: Trajectory Observability - Full multi-turn traces, not just outputs
Principle X: Cross-Functional Observability - PMs, domain experts, and AI engineers collaborate

When to use

After /evals-validate: Analyze failures and resolve them
Closing a development loop: Translate evaluation failure insights into rule or evaluator fixes
Reporting to stakeholders: Generate readable summaries for PMs and domain experts

When NOT to use

Evals not yet executed: Run /evals-validate first to generate results in evals/results/
Trivial tasks: Closed-loop analysis is overhead for simple features

Process

User Input

$ARGUMENTS

--focus AREA — Focus analysis on specific areas (e.g., security, quality, performance)
--dry-run — Analyze results and print report, but skip PR creation and local skill triggers

Execution Steps

Phase 1: Load Evaluation Results

Reads results JSON from evals/results/.
Extracts failure cases and full multi-turn conversation traces (including tool calls).

Phase 2: Failure Classification

Categorizes each failure trace:

Specification Failure: The agent was correct relative to its context, but the rule/directive was missing, ambiguous, or incorrect. Subclassify (EVAL-010):
- Mechanical: the gap is a fixed, checkable pattern (banned API, import shape, file-location, syntactic shape) → deterministic check territory
- Judgment gap: the gap needs intent, context, or cross-file judgement → context rule territory
Generalization Failure: The rule was correct, but the agent made a mistake anyway (hallucinated, missed a constraint, or grader lacked edge-case coverage).

Phase 3: Action Routing (Close the Loop)

For Mechanical Specification Failures: Fix/extend the existing binary grader or add a unit test that enforces the pattern. Do NOT propose a CDR for a mechanically-checkable gap — pay once for the check instead of re-deriving it per session.
For Judgment-gap Specification Failures: Automatically triggers local skill /team-learn with the failure trace as input. This creates new rule/persona/example CDRs in adlc branch drafts/cdr/ to fix the agent's behavior.
For Generalization Failures: Appends an evaluator backlog item to evals/results/evaluator_backlog.md detailing the needed grader edge-case updates.

Phase 4: Cross-Functional Insights & PR

Generates a stakeholder-specific report in evals/results/team_insights.md (tailored for PMs, domain experts, and AI engineers).
If git remote and gh CLI are available, commits rule/eval changes in team-ai-directives and opens a draft PR (uses team-learn logic under the hood).

Verification

Trajectory failure traces analyzed and classified
Mechanical specification failures routed to grader/unit-test fixes (deterministic checks); judgment-gap failures routed to /team-learn (proposes CDRs in adlc branch drafts/cdr/)
Generalization failures written to evals/results/evaluator_backlog.md
Stakeholder report evals/results/team_insights.md generated
Draft PR created in team-ai-directives (if applicable)
Final report summary presented with PR link and backlog details

Files (adlc-team-skills)

scripts

bash

setup-evals-analyze.sh 1.2 KB

#!/usr/bin/env bash
# setup-evals-analyze.sh — Setup for evals-init (self-contained)
set -euo pipefail

resolve_project_root() {
  local dir
  dir="$(pwd)"
  while [[ "$dir" != "/" ]]; do
    if [[ -d "${dir}/.adlc" ]]; then
      echo "$dir"
      return
    fi
    dir="$(dirname "$dir")"
  done
  git rev-parse --show-toplevel 2>/dev/null || pwd
}

resolve_team_ai_directives() {
  local project_root="$1"
  local td="${TEAM_AI_DIRECTIVES:-}"
  [[ -n "$td" ]] && { echo "$td"; return; }
  if [[ -f "${project_root}/.adlc/init-options.json" ]]; then
    td=$(python3 -c "
import json
try:
    with open('${project_root}/.adlc/init-options.json') as f:
        print(json.load(f).get('team_ai_directives', ''))
except Exception:
    print('')
" 2>/dev/null || true)
    [[ -n "$td" ]] && { echo "$td"; return; }
  fi
  echo "${project_root}/team-ai-directives"
}

resolve_branch() {
  git branch --show-current 2>/dev/null || echo "unknown"
}

PROJECT_ROOT=$(resolve_project_root)
TEAM_AI_DIRECTIVES=$(resolve_team_ai_directives "$PROJECT_ROOT")
BRANCH=$(resolve_branch)

python3 - "$PROJECT_ROOT" "$TEAM_AI_DIRECTIVES" "$BRANCH" << 'PY'
import json, sys
print(json.dumps({
  "REPO_ROOT": sys.argv[1],
  "TEAM_AI_DIRECTIVES": sys.argv[2],
  "BRANCH": sys.argv[3]
}))
PY

powershell
- setup-evals-analyze.ps1 1.3 KB · in bundle

SKILL.md 4.7 KB

---
name: evals-analyze
description: Use when evaluation results need triage and loop-closing — spec failures route to deterministic checks or context rules, generalization failures to the evaluator backlog.
disable-model-invocation: true
---

# evals-analyze

## What this skill does

Provides **cross-functional team elevation** and **closed-loop feedback** following **EDD Principle VIII** (Close the Production Loop) by deep-analyzing trajectory failure traces and routing them to correct resolution pathways.

**Output**:
1. **Trajectory Analysis** - Full multi-turn trace analysis with tool calls and context preservation (EDD Principle V)
2. **Failure Routing**:
   - **Specification Failures** (agent logic missing/ambiguous) → subclassified (EVAL-010):
     - **Mechanical** (fixed, checkable pattern) → fix/extend the existing grader or add a unit test — a deterministic check, not a CDR
     - **Judgment gap** (missing intent, ambiguity, needs context) → automatically triggers a local call to `team-learn` to propose new context rules in `adlc branch drafts/cdr/` to fix agent behavior
   - **Generalization Failures** (grader flawed or lacks edge-case coverage) → Appends evaluator backlog items to the project backlog for ongoing monitoring.
3. **Cross-Functional PR** - Creates a team-ai-directives PR with insights and rule updates (EDD Principle X)

**Key EDD Principles Applied**:
- **Principle VIII**: Close Production Loop - Spec failures → fix directives; Gen failures → evaluator backlog
- **Principle V**: Trajectory Observability - Full multi-turn traces, not just outputs
- **Principle X**: Cross-Functional Observability - PMs, domain experts, and AI engineers collaborate

## When to use

- **After `/evals-validate`**: Analyze failures and resolve them
- **Closing a development loop**: Translate evaluation failure insights into rule or evaluator fixes
- **Reporting to stakeholders**: Generate readable summaries for PMs and domain experts

## When NOT to use

- **Evals not yet executed**: Run `/evals-validate` first to generate results in `evals/results/`
- **Trivial tasks**: Closed-loop analysis is overhead for simple features

## Process

### User Input
```text
$ARGUMENTS
```
- `--focus AREA` — Focus analysis on specific areas (e.g., security, quality, performance)
- `--dry-run` — Analyze results and print report, but skip PR creation and local skill triggers

### Execution Steps

#### Phase 1: Load Evaluation Results
- Reads results JSON from `evals/results/`.
- Extracts failure cases and full multi-turn conversation traces (including tool calls).

#### Phase 2: Failure Classification
Categorizes each failure trace:
- **Specification Failure**: The agent was correct relative to its context, but the rule/directive was missing, ambiguous, or incorrect. Subclassify (EVAL-010):
  - **Mechanical**: the gap is a fixed, checkable pattern (banned API, import shape, file-location, syntactic shape) → deterministic check territory
  - **Judgment gap**: the gap needs intent, context, or cross-file judgement → context rule territory
- **Generalization Failure**: The rule was correct, but the agent made a mistake anyway (hallucinated, missed a constraint, or grader lacked edge-case coverage).

#### Phase 3: Action Routing (Close the Loop)
- **For Mechanical Specification Failures**: Fix/extend the existing binary grader or add a unit test that enforces the pattern. Do NOT propose a CDR for a mechanically-checkable gap — pay once for the check instead of re-deriving it per session.
- **For Judgment-gap Specification Failures**: Automatically triggers local skill `/team-learn` with the failure trace as input. This creates new rule/persona/example CDRs in `adlc branch drafts/cdr/` to fix the agent's behavior.
- **For Generalization Failures**: Appends an evaluator backlog item to `evals/results/evaluator_backlog.md` detailing the needed grader edge-case updates.

#### Phase 4: Cross-Functional Insights & PR
- Generates a stakeholder-specific report in `evals/results/team_insights.md` (tailored for PMs, domain experts, and AI engineers).
- If git remote and gh CLI are available, commits rule/eval changes in `team-ai-directives` and opens a draft PR (uses `team-learn` logic under the hood).

## Verification
- Trajectory failure traces analyzed and classified
- Mechanical specification failures routed to grader/unit-test fixes (deterministic checks); judgment-gap failures routed to `/team-learn` (proposes CDRs in `adlc branch drafts/cdr/`)
- Generalization failures written to `evals/results/evaluator_backlog.md`
- Stakeholder report `evals/results/team_insights.md` generated
- Draft PR created in team-ai-directives (if applicable)
- Final report summary presented with PR link and backlog details