Claude Skill

evals-clarify

Refine, cluster, and accept draft criteria into the published goldset. Isolates 20% holdout split and publishes goldset.md + goldset.json.

LLM Mart · 0 points · 9 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download tikalk-adlc-team-skills-skills_evals_evals-clarify-3035db2.zip · 3 KB
tikalk/adlc-team-skills 137 1 forks MIT Updated 2d ago
Part of tikalk/adlc-team-skills — 21 skills

Install

skills CLI npx skills add https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-clarify
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tikalk-adlc-team-skills@llmmart
Git git clone https://github.com/tikalk/adlc-team-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tikalk/adlc-team-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

evals-clarify

What this skill does

Conducts axial coding following EDD Principles III & IX to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.

Output:

  1. Clustered Criteria - Related patterns grouped into coherent evaluation themes
  2. Adversarial Examples - Generated attack scenarios and edge cases for robustness
  3. Published Goldset - Accepted criteria in evals/{system}/goldset.md with full documentation
  4. Holdout Dataset - Reserved test set (20%) for unbiased evaluation validation
  5. JSON Configuration - Auto-generated goldset.json for system consumption
  6. Auto-handoff to /evals-implement for grader generation

Key EDD Principles Applied:

  • Principle III: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
  • Principle IX: Test Data as Code - Adversarial generation, holdout splits, version control
  • Principle II: Binary Pass/Fail - Maintain strict binary evaluation throughout
  • Principle I: Spec-Driven Contracts - Criteria validate spec compliance

When to use

  • After /evals-specify: Refine and accept draft criteria into goldset
  • Dataset maintenance: Balance pass/fail examples or add adversarial cases
  • Adding holdout split: Isolate validation data from training data

When NOT to use

  • No draft criteria exist: Run /evals-specify to discover patterns first
  • Grader generation: Use /evals-implement to convert accepted goldset into code

Process

User Input

$ARGUMENTS
  • --accept IDS — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
  • --merge IDS — Merge related criteria (e.g., "EVAL-001+EVAL-002")
  • --split ID — Split complex criterion into multiple focused criteria
  • --holdout-ratio RATIO — Holdout percentage (default: 0.2, range: 0.1-0.3)

Execution Steps

Step 0: Detect Lightweight Draft Format

Check if the draft being reviewed uses the lightweight draft template (indicated by presence of type, evidence, source, revisit-when fields in frontmatter and ## Rejected Alternatives / ## Reason body sections without the full formal template sections).

If lightweight:

  1. Read the draft's captured fields (Context, Decision, Rejected Alternatives, Reason)
  2. Transform to full formal template:
    • ADR: full MADR format with Decision Drivers, Considered Options, Pros/Cons, Constitution Alignment, Related ADRs
    • PDR: full PDR format with Market Forces, Consequences, Alternatives Considered, Links
    • ChDR: full ChDR format with Issue Links, Commits, Consequences, Evidence
    • CDR: full CDR format with Context Type, Target Module, Descriptor, Evidence
    • EVAL: full eval format with Error Analysis, Pass/Fail Examples, Implementation Notes
  3. Enrich from session context (add details the lightweight draft may have omitted)
  4. Present the enriched draft for review

If already full format, proceed with normal review.

Phase 1: Axial Coding & Clustering

  • Group related draft patterns into coherent themes.
  • Resolve any overlaps or duplicate criteria.

Phase 2: Refinement & Adversarial Generation

  • Generate 3-5 adversarial (attack) examples per criterion to test robustness.
  • Balance pass/fail examples (~50/50 ratio).

Phase 3: Holdout Isolation

  • Isolate exactly 20% of examples as a reserved holdout set (saved to .adlc/memory/evals/holdout.json).
  • Ensure holdout set is never used in implementation or training.

Phase 4: Publish Goldset

  • Copy accepted drafts to .adlc/memory/evals/ and update status to accepted.
  • Compile published goldset to evals/{system}/goldset.md (human-readable) and evals/{system}/goldset.json (machine-readable).

Phase 5: Auto-Handoff

Trigger /evals-implement to generate code.

Verification

  • Accepted drafts stored in .adlc/memory/evals/EVAL-*.md
  • evals/{system}/goldset.md and goldset.json exist
  • Holdout set .adlc/memory/evals/holdout.json isolated and populated
  • All criteria are strictly binary (no confidence scores or Likert scales)
  • Handover summary lists accepted criteria and adversarial counts
Files (adlc-team-skills)
  • scripts
    • bash
      • setup-evals-clarify.sh 1.2 KB 
        #!/usr/bin/env bash
# setup-evals-clarify.sh — Setup for evals-init (self-contained)
set -euo pipefail

resolve_project_root() {
  local dir
  dir="$(pwd)"
  while [[ "$dir" != "/" ]]; do
    if [[ -d "${dir}/.adlc" ]]; then
      echo "$dir"
      return
    fi
    dir="$(dirname "$dir")"
  done
  git rev-parse --show-toplevel 2>/dev/null || pwd
}

resolve_team_ai_directives() {
  local project_root="$1"
  local td="${TEAM_AI_DIRECTIVES:-}"
  [[ -n "$td" ]] && { echo "$td"; return; }
  if [[ -f "${project_root}/.adlc/init-options.json" ]]; then
    td=$(python3 -c "
import json
try:
    with open('${project_root}/.adlc/init-options.json') as f:
        print(json.load(f).get('team_ai_directives', ''))
except Exception:
    print('')
" 2>/dev/null || true)
    [[ -n "$td" ]] && { echo "$td"; return; }
  fi
  echo "${project_root}/team-ai-directives"
}

resolve_branch() {
  git branch --show-current 2>/dev/null || echo "unknown"
}

PROJECT_ROOT=$(resolve_project_root)
TEAM_AI_DIRECTIVES=$(resolve_team_ai_directives "$PROJECT_ROOT")
BRANCH=$(resolve_branch)

python3 - "$PROJECT_ROOT" "$TEAM_AI_DIRECTIVES" "$BRANCH" << 'PY'
import json, sys
print(json.dumps({
  "REPO_ROOT": sys.argv[1],
  "TEAM_AI_DIRECTIVES": sys.argv[2],
  "BRANCH": sys.argv[3]
}))
PY
    • powershell
      • setup-evals-clarify.ps1 1.3 KB · in bundle
  • SKILL.md 4.4 KB 
    ---
name: evals-clarify
description: Use when draft eval criteria need refining, clustering, and acceptance into the published goldset with an isolated holdout split (goldset.md + goldset.json).
disable-model-invocation: true
---

# evals-clarify

## What this skill does

Conducts **axial coding** following **EDD Principles III & IX** to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.

**Output**:
1. **Clustered Criteria** - Related patterns grouped into coherent evaluation themes
2. **Adversarial Examples** - Generated attack scenarios and edge cases for robustness
3. **Published Goldset** - Accepted criteria in `evals/{system}/goldset.md` with full documentation
4. **Holdout Dataset** - Reserved test set (20%) for unbiased evaluation validation
5. **JSON Configuration** - Auto-generated `goldset.json` for system consumption
6. **Auto-handoff** to `/evals-implement` for grader generation

**Key EDD Principles Applied**:
- **Principle III**: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
- **Principle IX**: Test Data as Code - Adversarial generation, holdout splits, version control
- **Principle II**: Binary Pass/Fail - Maintain strict binary evaluation throughout
- **Principle I**: Spec-Driven Contracts - Criteria validate spec compliance

## When to use

- **After `/evals-specify`**: Refine and accept draft criteria into goldset
- **Dataset maintenance**: Balance pass/fail examples or add adversarial cases
- **Adding holdout split**: Isolate validation data from training data

## When NOT to use

- **No draft criteria exist**: Run `/evals-specify` to discover patterns first
- **Grader generation**: Use `/evals-implement` to convert accepted goldset into code

## Process

### User Input
```text
$ARGUMENTS
```
- `--accept IDS` — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
- `--merge IDS` — Merge related criteria (e.g., "EVAL-001+EVAL-002")
- `--split ID` — Split complex criterion into multiple focused criteria
- `--holdout-ratio RATIO` — Holdout percentage (default: 0.2, range: 0.1-0.3)

### Execution Steps

#### Step 0: Detect Lightweight Draft Format

Check if the draft being reviewed uses the lightweight draft template (indicated
by presence of `type`, `evidence`, `source`, `revisit-when` fields in frontmatter
and `## Rejected Alternatives` / `## Reason` body sections without the full
formal template sections).

If lightweight:
1. Read the draft's captured fields (Context, Decision, Rejected Alternatives, Reason)
2. Transform to full formal template:
   - ADR: full MADR format with Decision Drivers, Considered Options, Pros/Cons, Constitution Alignment, Related ADRs
   - PDR: full PDR format with Market Forces, Consequences, Alternatives Considered, Links
   - ChDR: full ChDR format with Issue Links, Commits, Consequences, Evidence
   - CDR: full CDR format with Context Type, Target Module, Descriptor, Evidence
   - EVAL: full eval format with Error Analysis, Pass/Fail Examples, Implementation Notes
3. Enrich from session context (add details the lightweight draft may have omitted)
4. Present the enriched draft for review

If already full format, proceed with normal review.

#### Phase 1: Axial Coding & Clustering
- Group related draft patterns into coherent themes.
- Resolve any overlaps or duplicate criteria.

#### Phase 2: Refinement & Adversarial Generation
- Generate 3-5 adversarial (attack) examples per criterion to test robustness.
- Balance pass/fail examples (~50/50 ratio).

#### Phase 3: Holdout Isolation
- Isolate exactly 20% of examples as a reserved holdout set (saved to `.adlc/memory/evals/holdout.json`).
- Ensure holdout set is never used in implementation or training.

#### Phase 4: Publish Goldset
- Copy accepted drafts to `.adlc/memory/evals/` and update status to `accepted`.
- Compile published goldset to `evals/{system}/goldset.md` (human-readable) and `evals/{system}/goldset.json` (machine-readable).

#### Phase 5: Auto-Handoff
Trigger `/evals-implement` to generate code.

## Verification
- Accepted drafts stored in `.adlc/memory/evals/EVAL-*.md`
- `evals/{system}/goldset.md` and `goldset.json` exist
- Holdout set `.adlc/memory/evals/holdout.json` isolated and populated
- All criteria are strictly binary (no confidence scores or Likert scales)
- Handover summary lists accepted criteria and adversarial counts

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related

Claude Prompt
Tighten-my-prose editing prompt
Claude Skill
Adversarial code-review skill for Claude
Claude Workflow
Deep-research workflow (fan-out + verify)
Claude Skill
Conventional-commit writer for Claude