{"slug":"evals-specify","title":"evals-specify","summary":"Extract eval criteria from product specs and production failure traces (bottom-up error analysis). Writes proposed criteria to .adlc/drafts/evals/.","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-06T17:21:37.116011Z","repo":{"url":"https://github.com/tikalk/adlc-team-skills","stars":139,"forks":1,"license":"MIT","updatedAt":"2026-09-30T13:59:46Z"},"bodyHtml":"<hr>\n<h2>name: evals-specify\ndescription: Extract eval criteria from product specs and production failure traces (bottom-up error analysis). Writes proposed criteria to .adlc/drafts/evals/.\ndisable-model-invocation: true</h2>\n<h1>evals-specify</h1>\n<h2>What this skill does</h2>\n<p>Conducts <strong>bottom-up error analysis</strong> following <strong>EDD Principles III &amp; IX</strong> (Error Analysis &amp; Test Data as Code) to discover and document draft evaluation criteria from human observation of system failures.</p>\n<p><strong>Output</strong>:</p>\n<ol>\n<li><strong>Draft Eval Records</strong> - Individual <code>EVAL-*.md</code> files in <code>.adlc/drafts/evals/</code> with open coding notes</li>\n<li><strong>Error Pattern Documentation</strong> - Bottom-up failure taxonomy from actual traces</li>\n<li><strong>Pass/Fail Examples</strong> - Real examples that should pass/fail each criterion</li>\n<li><strong>Auto-handoff</strong> to <code>/evals-clarify</code> for axial coding and clustering</li>\n</ol>\n<p><strong>Key EDD Principles Applied</strong>:</p>\n<ul>\n<li><strong>Principle III</strong>: Error Analysis &amp; Pattern Discovery - Open coding → failure taxonomy</li>\n<li><strong>Principle IX</strong>: Test Data as Code - Dataset planning and coverage analysis</li>\n<li><strong>Principle II</strong>: Binary Pass/Fail - Maintain strict binary pass/fail conditions</li>\n<li><strong>Principle V</strong>: Trajectory Observability - Track full multi-turn conversation traces</li>\n</ul>\n<h2>When to use</h2>\n<ul>\n<li><strong>Starting evaluation development</strong>: No existing criteria, need discovery from failure logs</li>\n<li><strong>Production incident analysis</strong>: Recent failures require systematic analysis</li>\n<li><strong>Quality assessment</strong>: Discovering and codifying boundary conditions from failures</li>\n</ul>\n<h2>When NOT to use</h2>\n<ul>\n<li><strong>No failure traces/specs</strong>: Generate synthetic traces first, or use <code>/evals-init</code> to set up security baselines</li>\n<li><strong>Known criteria already exist</strong>: Use <code>/evals-clarify</code> to refine or <code>/evals-implement</code> to generate code</li>\n</ul>\n<h2>Process</h2>\n<h3>User Input</h3>\n<pre><code>$ARGUMENTS\n</code></pre>\n<p>Treat user input as specific failure areas or error patterns to analyze (e.g., \"authentication bypass\", \"RAG irrelevant results\").</p>\n<ul>\n<li><code>--traces N</code> — Number of traces to analyze (default: 20, min for theoretical saturation)</li>\n<li><code>--source SOURCE</code> — Trace source location (e.g., logs, support tickets)</li>\n</ul>\n<h3>Execution Steps</h3>\n<h4>Phase 1: Open Coding Analysis</h4>\n<ul>\n<li>Reviews the user-provided failure logs or spec requirements.</li>\n<li>Conducts open coding of traces to discover recurring failure patterns (EDD Principle III).</li>\n<li>Identifies: core problem, causal conditions, and consequences.</li>\n</ul>\n<h4>Phase 2: Create Draft Criteria</h4>\n<p>Group patterns into draft criteria. For each:</p>\n<ul>\n<li>Define strict <strong>Pass Condition</strong> (observable, binary yes/no)</li>\n<li>Define strict <strong>Fail Condition</strong> (observable, binary yes/no)</li>\n<li>Document real pass/fail examples directly from traces</li>\n</ul>\n<h4>Phase 3: Create Draft Files</h4>\n<ul>\n<li>Copy <code>skills/evals/evals-templates/eval-criterion-template.md</code> to <code>.adlc/drafts/evals/EVAL-{NNN}.md</code>.</li>\n<li>Populate metadata and error analysis notes.</li>\n<li>Regenerate index at <code>.adlc/drafts/evals/evals.md</code>.</li>\n</ul>\n<h4>Phase 4: Auto-Handoff</h4>\n<p>Trigger <code>/evals-clarify</code> for axial coding and clustering.</p>\n<h2>Verification</h2>\n<ul>\n<li>Draft files created at <code>.adlc/drafts/evals/EVAL-*.md</code></li>\n<li>Index file <code>.adlc/drafts/evals/evals.md</code> updated with draft summaries</li>\n<li>Each draft contains: status \"draft\", pass/fail conditions, trace sources, and concrete examples</li>\n<li>Auto-handoff context produced with list of created drafts</li>\n</ul>\n","files":[{"path":"scripts/bash/setup-evals-specify.sh","sizeBytes":1272,"isText":true},{"path":"scripts/powershell/setup-evals-specify.ps1","sizeBytes":1359,"isText":false},{"path":"SKILL.md","sizeBytes":3668,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-23T13:51:12.289941Z","sha256":"046F154B1CBFBF3AE2A82B95E14AF024D0E70EE97A42DA7297333683BA830C44","sizeBytes":3293},"review":null,"source":{"repositoryUrl":"https://github.com/tikalk/adlc-team-skills","path":"skills/evals/evals-specify","license":"MIT","commit":"249c1df30f916150d87a6f734d2bd2cb308cae8d","subtreeSha":"92357B2FA80A73F033F39A5478ECF60B3293A6C6BB483E06A07EAF3511B8945F","lastSyncedAt":"2026-09-30T15:23:42.806745Z"},"reviewedAt":"2026-09-23T14:00:10.426498Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-specify"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tikalk-adlc-team-skills@llmmart"},{"target":"git","command":"git clone https://github.com/tikalk/adlc-team-skills.git"}]}