Claude Skill

evals-validate

Run evaluations and validate evaluator quality (SLA compliance, TPR/TNR, statistical accuracy). Executes PromptFoo or pytest DeepEval.

LLM Mart · 0 points · 7 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download tikalk-adlc-team-skills-skills_evals_evals-validate-3035db2.zip · 3 KB

tikalk/adlc-team-skills 137 1 forks MIT Updated 2d ago

Part of tikalk/adlc-team-skills — 21 skills

Install

skills CLI npx skills add https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-validate

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tikalk-adlc-team-skills@llmmart

Git git clone https://github.com/tikalk/adlc-team-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tikalk/adlc-team-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

evals-validate

What this skill does

Conducts comprehensive validation of the implemented evaluation system following EDD principles to ensure production readiness through statistical analysis, performance verification, and quality assurance.

Output:

Statistical Validation - TPR/TNR analysis, accuracy metrics, confidence intervals
Performance Validation - SLA compliance verification for evaluation pyramid tiers
Quality Assurance - Goldset integrity, example balance, coverage analysis
Holdout Dataset Validation - Unbiased accuracy assessment on reserved test set
Auto-handoff to /evals-analyze for closed loop trajectory analysis

Key EDD Principles Applied:

Principle IV: Evaluation Pyramid - Tier performance SLA validation (Tier 1 <30s, Tier 2 <5min)
Principle II: Binary Pass/Fail - Statistical compliance verification
Principle IX: Test Data as Code - Holdout dataset validation integrity
Principle III: Error Analysis - Pattern stability validation

When to use

After /evals-implement: Execute the evaluation suite and measure quality
CI/CD Pipeline gate: Run evaluations before release to ensure no regressions
Periodic audit: Verify evaluator accuracy on holdout data to check for model drift

When NOT to use

Evaluator not generated: Run /evals-implement to build grader files first
Analysing failure traces: Use /evals-analyze to extract deep insights from run results

Process

User Input

$ARGUMENTS

--holdout-only — Validate only on holdout dataset (unbiased validation)
--performance-only — Skip statistical analysis, focus on SLA compliance
--metrics METRICS — Specific metrics to validate (tpr, tnr, accuracy, performance)

Execution Steps

Phase 1: Execute Evaluations

Runs the underlying framework CLI directly:

PromptFoo: npx promptfoo eval --config evals/promptfoo/config.js
DeepEval: pytest evals/deepeval/ -v or python evals/deepeval/config.py

Phase 2: Compute Statistical Validation

Parse generated results JSON from evals/results/.
Calculate True Positive Rate (TPR) and True Negative Rate (TNR).
Calculate overall accuracy with 95% confidence intervals.
Ensure no Likert scales or numerical scores leak into results.

Phase 3: SLA Compliance Check

Measure execution times for Tier 1 and Tier 2.
Verify Tier 1 completes under 30 seconds.
Verify Tier 2 completes under 5 minutes.
Check headroom analysis (SLA budget consumed).

Phase 4: Write Validation Report

Write validation results to evals/results/validation_report.md.
Include pass/fail counts, TPR/TNR table, SLA timings, and holdout set results.

Phase 5: Auto-Handoff

Trigger /evals-analyze to close the loop.

Verification

Evaluation execution successfully completed with results JSON written to evals/results/
evals/results/validation_report.md created with TPR/TNR and SLA metrics
Statistical metrics calculated with confidence intervals
Headroom and SLA compliance verified
Handover summary lists results and validation report path

Files (adlc-team-skills)

scripts

bash

setup-evals-validate.sh 1.2 KB

#!/usr/bin/env bash
# setup-evals-validate.sh — Setup for evals-init (self-contained)
set -euo pipefail

resolve_project_root() {
  local dir
  dir="$(pwd)"
  while [[ "$dir" != "/" ]]; do
    if [[ -d "${dir}/.adlc" ]]; then
      echo "$dir"
      return
    fi
    dir="$(dirname "$dir")"
  done
  git rev-parse --show-toplevel 2>/dev/null || pwd
}

resolve_team_ai_directives() {
  local project_root="$1"
  local td="${TEAM_AI_DIRECTIVES:-}"
  [[ -n "$td" ]] && { echo "$td"; return; }
  if [[ -f "${project_root}/.adlc/init-options.json" ]]; then
    td=$(python3 -c "
import json
try:
    with open('${project_root}/.adlc/init-options.json') as f:
        print(json.load(f).get('team_ai_directives', ''))
except Exception:
    print('')
" 2>/dev/null || true)
    [[ -n "$td" ]] && { echo "$td"; return; }
  fi
  echo "${project_root}/team-ai-directives"
}

resolve_branch() {
  git branch --show-current 2>/dev/null || echo "unknown"
}

PROJECT_ROOT=$(resolve_project_root)
TEAM_AI_DIRECTIVES=$(resolve_team_ai_directives "$PROJECT_ROOT")
BRANCH=$(resolve_branch)

python3 - "$PROJECT_ROOT" "$TEAM_AI_DIRECTIVES" "$BRANCH" << 'PY'
import json, sys
print(json.dumps({
  "REPO_ROOT": sys.argv[1],
  "TEAM_AI_DIRECTIVES": sys.argv[2],
  "BRANCH": sys.argv[3]
}))
PY

powershell
- setup-evals-validate.ps1 1.3 KB · in bundle

SKILL.md 3.3 KB

---
name: evals-validate
description: Use when a goldset with graders is ready to run — executes the evaluation pyramid and validates evaluator quality (SLA compliance, TPR/TNR, statistical accuracy).
disable-model-invocation: true
---

# evals-validate

## What this skill does

Conducts **comprehensive validation** of the implemented evaluation system following **EDD principles** to ensure production readiness through statistical analysis, performance verification, and quality assurance.

**Output**:
1. **Statistical Validation** - TPR/TNR analysis, accuracy metrics, confidence intervals
2. **Performance Validation** - SLA compliance verification for evaluation pyramid tiers
3. **Quality Assurance** - Goldset integrity, example balance, coverage analysis
4. **Holdout Dataset Validation** - Unbiased accuracy assessment on reserved test set
5. **Auto-handoff** to `/evals-analyze` for closed loop trajectory analysis

**Key EDD Principles Applied**:
- **Principle IV**: Evaluation Pyramid - Tier performance SLA validation (Tier 1 <30s, Tier 2 <5min)
- **Principle II**: Binary Pass/Fail - Statistical compliance verification
- **Principle IX**: Test Data as Code - Holdout dataset validation integrity
- **Principle III**: Error Analysis - Pattern stability validation

## When to use

- **After `/evals-implement`**: Execute the evaluation suite and measure quality
- **CI/CD Pipeline gate**: Run evaluations before release to ensure no regressions
- **Periodic audit**: Verify evaluator accuracy on holdout data to check for model drift

## When NOT to use

- **Evaluator not generated**: Run `/evals-implement` to build grader files first
- **Analysing failure traces**: Use `/evals-analyze` to extract deep insights from run results

## Process

### User Input
```text
$ARGUMENTS
```
- `--holdout-only` — Validate only on holdout dataset (unbiased validation)
- `--performance-only` — Skip statistical analysis, focus on SLA compliance
- `--metrics METRICS` — Specific metrics to validate (tpr, tnr, accuracy, performance)

### Execution Steps

#### Phase 1: Execute Evaluations
Runs the underlying framework CLI directly:
- PromptFoo: `npx promptfoo eval --config evals/promptfoo/config.js`
- DeepEval: `pytest evals/deepeval/ -v` or `python evals/deepeval/config.py`

#### Phase 2: Compute Statistical Validation
- Parse generated results JSON from `evals/results/`.
- Calculate True Positive Rate (TPR) and True Negative Rate (TNR).
- Calculate overall accuracy with 95% confidence intervals.
- Ensure no Likert scales or numerical scores leak into results.

#### Phase 3: SLA Compliance Check
- Measure execution times for Tier 1 and Tier 2.
- Verify Tier 1 completes under 30 seconds.
- Verify Tier 2 completes under 5 minutes.
- Check headroom analysis (SLA budget consumed).

#### Phase 4: Write Validation Report
- Write validation results to `evals/results/validation_report.md`.
- Include pass/fail counts, TPR/TNR table, SLA timings, and holdout set results.

#### Phase 5: Auto-Handoff
Trigger `/evals-analyze` to close the loop.

## Verification
- Evaluation execution successfully completed with results JSON written to `evals/results/`
- `evals/results/validation_report.md` created with TPR/TNR and SLA metrics
- Statistical metrics calculated with confidence intervals
- Headroom and SLA compliance verified
- Handover summary lists results and validation report path