Claude Skill

evals-specify

Extract eval criteria from product specs and production failure traces (bottom-up error analysis). Writes proposed criteria to .adlc/drafts/evals/.

LLM Mart · 0 points · 7 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download tikalk-adlc-team-skills-skills_evals_evals-specify-3035db2.zip · 3 KB

tikalk/adlc-team-skills 137 1 forks MIT Updated 2d ago

Part of tikalk/adlc-team-skills — 21 skills

Install

skills CLI npx skills add https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-specify

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tikalk-adlc-team-skills@llmmart

Git git clone https://github.com/tikalk/adlc-team-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tikalk/adlc-team-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

evals-specify

What this skill does

Conducts bottom-up error analysis following EDD Principles III & IX (Error Analysis & Test Data as Code) to discover and document draft evaluation criteria from human observation of system failures.

Output:

Draft Eval Records - Individual EVAL-*.md files in .adlc/drafts/evals/ with open coding notes
Error Pattern Documentation - Bottom-up failure taxonomy from actual traces
Pass/Fail Examples - Real examples that should pass/fail each criterion
Auto-handoff to /evals-clarify for axial coding and clustering

Key EDD Principles Applied:

Principle III: Error Analysis & Pattern Discovery - Open coding → failure taxonomy
Principle IX: Test Data as Code - Dataset planning and coverage analysis
Principle II: Binary Pass/Fail - Maintain strict binary pass/fail conditions
Principle V: Trajectory Observability - Track full multi-turn conversation traces

When to use

Note: Routine decision capture is handled by team-boot's continuous capture mechanism, which writes lightweight drafts directly to .adlc/drafts/. This skill is for interactive deep-dive exploration — when you want guided trade-off analysis, multi-option comparison, or structured decision facilitation before documenting.

Starting evaluation development: No existing criteria, need discovery from failure logs
Production incident analysis: Recent failures require systematic analysis
Quality assessment: Discovering and codifying boundary conditions from failures

When NOT to use

No failure traces/specs: Generate synthetic traces first, or use /evals-init to set up security baselines
Known criteria already exist: Use /evals-clarify to refine or /evals-implement to generate code

Process

User Input

$ARGUMENTS

Treat user input as specific failure areas or error patterns to analyze (e.g., "authentication bypass", "RAG irrelevant results").

--traces N — Number of traces to analyze (default: 20, min for theoretical saturation)
--source SOURCE — Trace source location (e.g., logs, support tickets)

Execution Steps

Phase 1: Open Coding Analysis

Reviews the user-provided failure logs or spec requirements.
Conducts open coding of traces to discover recurring failure patterns (EDD Principle III).
Identifies: core problem, causal conditions, and consequences.

Phase 2: Create Draft Criteria

Group patterns into draft criteria. For each:

Define strict Pass Condition (observable, binary yes/no)
Define strict Fail Condition (observable, binary yes/no)
Document real pass/fail examples directly from traces

Phase 3: Create Draft Files

Copy skills/evals/evals-templates/eval-criterion-template.md to .adlc/drafts/evals/EVAL-{NNN}.md.
Populate metadata and error analysis notes.
Regenerate index at .adlc/drafts/evals/evals.md.

Phase 4: Auto-Handoff

Trigger /evals-clarify for axial coding and clustering.

Verification

Draft files created at .adlc/drafts/evals/EVAL-*.md
Index file .adlc/drafts/evals/evals.md updated with draft summaries
Each draft contains: status "draft", pass/fail conditions, trace sources, and concrete examples
Auto-handoff context produced with list of created drafts

Files (adlc-team-skills)

scripts

bash

setup-evals-specify.sh 1.2 KB

#!/usr/bin/env bash
# setup-evals-specify.sh — Setup for evals-init (self-contained)
set -euo pipefail

resolve_project_root() {
  local dir
  dir="$(pwd)"
  while [[ "$dir" != "/" ]]; do
    if [[ -d "${dir}/.adlc" ]]; then
      echo "$dir"
      return
    fi
    dir="$(dirname "$dir")"
  done
  git rev-parse --show-toplevel 2>/dev/null || pwd
}

resolve_team_ai_directives() {
  local project_root="$1"
  local td="${TEAM_AI_DIRECTIVES:-}"
  [[ -n "$td" ]] && { echo "$td"; return; }
  if [[ -f "${project_root}/.adlc/init-options.json" ]]; then
    td=$(python3 -c "
import json
try:
    with open('${project_root}/.adlc/init-options.json') as f:
        print(json.load(f).get('team_ai_directives', ''))
except Exception:
    print('')
" 2>/dev/null || true)
    [[ -n "$td" ]] && { echo "$td"; return; }
  fi
  echo "${project_root}/team-ai-directives"
}

resolve_branch() {
  git branch --show-current 2>/dev/null || echo "unknown"
}

PROJECT_ROOT=$(resolve_project_root)
TEAM_AI_DIRECTIVES=$(resolve_team_ai_directives "$PROJECT_ROOT")
BRANCH=$(resolve_branch)

python3 - "$PROJECT_ROOT" "$TEAM_AI_DIRECTIVES" "$BRANCH" << 'PY'
import json, sys
print(json.dumps({
  "REPO_ROOT": sys.argv[1],
  "TEAM_AI_DIRECTIVES": sys.argv[2],
  "BRANCH": sys.argv[3]
}))
PY

powershell
- setup-evals-specify.ps1 1.3 KB · in bundle

SKILL.md 3.6 KB

---
name: evals-specify
description: Use when you want guided bottom-up error analysis, structured failure taxonomy discovery, or comprehensive trace coding before documenting eval criteria. Optional for routine capture — team-boot writes lightweight eval drafts directly.
disable-model-invocation: true
---

# evals-specify

## What this skill does

Conducts **bottom-up error analysis** following **EDD Principles III & IX** (Error Analysis & Test Data as Code) to discover and document draft evaluation criteria from human observation of system failures.

**Output**:
1. **Draft Eval Records** - Individual `EVAL-*.md` files in `.adlc/drafts/evals/` with open coding notes
2. **Error Pattern Documentation** - Bottom-up failure taxonomy from actual traces
3. **Pass/Fail Examples** - Real examples that should pass/fail each criterion
4. **Auto-handoff** to `/evals-clarify` for axial coding and clustering

**Key EDD Principles Applied**:
- **Principle III**: Error Analysis & Pattern Discovery - Open coding → failure taxonomy
- **Principle IX**: Test Data as Code - Dataset planning and coverage analysis
- **Principle II**: Binary Pass/Fail - Maintain strict binary pass/fail conditions
- **Principle V**: Trajectory Observability - Track full multi-turn conversation traces

## When to use

> **Note**: Routine decision capture is handled by team-boot's continuous
> capture mechanism, which writes lightweight drafts directly to
> `.adlc/drafts/`. This skill is for **interactive deep-dive exploration**
> — when you want guided trade-off analysis, multi-option comparison, or
> structured decision facilitation before documenting.

- **Starting evaluation development**: No existing criteria, need discovery from failure logs
- **Production incident analysis**: Recent failures require systematic analysis
- **Quality assessment**: Discovering and codifying boundary conditions from failures

## When NOT to use

- **No failure traces/specs**: Generate synthetic traces first, or use `/evals-init` to set up security baselines
- **Known criteria already exist**: Use `/evals-clarify` to refine or `/evals-implement` to generate code

## Process

### User Input
```text
$ARGUMENTS
```
Treat user input as specific failure areas or error patterns to analyze (e.g., "authentication bypass", "RAG irrelevant results").
- `--traces N` — Number of traces to analyze (default: 20, min for theoretical saturation)
- `--source SOURCE` — Trace source location (e.g., logs, support tickets)

### Execution Steps

#### Phase 1: Open Coding Analysis
- Reviews the user-provided failure logs or spec requirements.
- Conducts open coding of traces to discover recurring failure patterns (EDD Principle III).
- Identifies: core problem, causal conditions, and consequences.

#### Phase 2: Create Draft Criteria
Group patterns into draft criteria. For each:
- Define strict **Pass Condition** (observable, binary yes/no)
- Define strict **Fail Condition** (observable, binary yes/no)
- Document real pass/fail examples directly from traces

#### Phase 3: Create Draft Files
- Copy `skills/evals/evals-templates/eval-criterion-template.md` to `.adlc/drafts/evals/EVAL-{NNN}.md`.
- Populate metadata and error analysis notes.
- Regenerate index at `.adlc/drafts/evals/evals.md`.

#### Phase 4: Auto-Handoff
Trigger `/evals-clarify` for axial coding and clustering.

## Verification
- Draft files created at `.adlc/drafts/evals/EVAL-*.md`
- Index file `.adlc/drafts/evals/evals.md` updated with draft summaries
- Each draft contains: status "draft", pass/fail conditions, trace sources, and concrete examples
- Auto-handoff context produced with list of created drafts