Claude Skill

skill-evaluation

Use this skill when designing, running, interpreting, or reporting Agent Skill evaluations, selecting cases or judges, and analyzing trigger, benchmark, or regression evidence; triggers include Skill evaluation and evaluation design.

LLM Mart · 0 points · 0 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download naodeng-awesome-qa-skills-skills_en_skill-engineering_skill-evaluation-c44b892.zip · 5 KB

naodeng/awesome-qa-skills 230 31 forks Updated 3d ago

Part of naodeng/awesome-qa-skills — 97 skills

Install

skills CLI npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/skill-engineering/skill-evaluation

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install naodeng-awesome-qa-skills@llmmart

Git git clone https://github.com/naodeng/awesome-qa-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole naodeng/awesome-qa-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Skill Evaluation

When to use

Design realistic eval cases, positive/negative triggers, or regression cases for a Skill.
Validate or run evaluations with skill-up and interpret results and limitations.
Select deterministic, script, or semantic judges, or distinguish a benchmark from a version regression.

Output format options

Default to a concise Markdown evaluation report with tables for case results and evidence states.
Use JSON only when a downstream script needs machine-readable case results; keep the same evidence vocabulary and limitations.

How to use

Read the Skill contract, existing evals/, historical failures, and the current change scope.
Identify critical behavior and evidence dimensions: Outcome, Process, Style/Quality, and Efficiency; select only meaningful dimensions.
Design HAPPY, INCOMPLETE, EXPLICIT_TRIGGER, IMPLICIT_TRIGGER, CONTEXTUAL_TRIGGER, NEGATIVE_TRIGGER, BOUNDARY, or REGRESSION cases.
Prefer deterministic rule_based; use script for executable artifacts; use a calibrated agent_judge only when semantic judgment is necessary.
Run skill-up validate; run skill-up run or the local trace runner only with authorized credentials and targets. Record run metadata, traces, judges, artifacts, and limitations.
Check Eval validity and distinguish Skill Defect, Eval Defect, Infrastructure Defect, and Unknown.
Report results, evidence states, benchmark/regression conclusions, blocked checks, and recommended actions; add a regression case only after a real failure has a confirmed root cause.

Constraints

skill-up is the primary Eval Engine. Trace checks are a deep-evidence layer; do not create another Engine, Judge, Benchmark, or Quality Score.
A similar output is not observed trigger evidence; missing skill.selection trace evidence is BLOCKED.
skill-up validate is not runtime semantic validation. Static checks, CLI smoke, Project Done, and one semantic observation cannot be promoted automatically to release or business claims.
Do not modify the Skill or enter an unlimited optimization loop. Keep Benchmark (with/without Skill) separate from Version Regression (previous/current).
Use unknown for unknown values; preserve NOT_RUN, UNASSESSED, BLOCKED, or INSUFFICIENT_EVIDENCE when evidence is incomplete.

Reference files

Read prompts/skill-evaluation.md for the complete execution and report contract.
Read the target Skill's evals/ and its fixtures before selecting a judge.
Use repository governance contracts and trace rules as optional deep references, never as private dependencies of a copied Skill.

Common pitfalls

Treating skill-up validate or a dry-run as proof of runtime or semantic effectiveness.
Calling a missing trigger event a negative result instead of BLOCKED.
Mixing with/without Skill benchmarks with previous/current version regression.

Best practices

Start with the smallest deterministic case that demonstrates the intended behavior.
Record the exact run identity, judge, environment, unavailable evidence, and limitations.
Convert only confirmed real failures into regression cases, and preserve the original evidence.

Progressive disclosure

Read prompts/skill-evaluation.md before producing the report.
Read the target Skill's own evals/ first, then load fixtures, examples, and scripts as needed.
Repository Evaluation Contract and local trace rules are optional deep references; an independently installed Skill must not depend on their private files.

Pre-delivery checklist

Every conclusion maps to a case, judge, and evidence state
Run metadata, environment, model, and unexecuted checks are explicit
Skill/Eval/Infrastructure/Unknown classifications are not conflated
Trigger, benchmark, regression, and Quality Score boundaries are clear

Files (awesome-qa-skills)

agents

openai.yaml 391 B

version: 1
metadata:
  key: "skill-evaluation"
interface:
  display_name: "Skill Evaluation"
  short_description: "Design, run, interpret, and report Skill evaluations while separating trigger, benchmark, regression, and evidence limits."
  default_prompt: "Use the skill-evaluation skill to design and report an evidence-bounded Skill evaluation."
policy:
  allow_implicit_invocation: true

evals

cases

basic-success.yaml 622 B

id: basic-success
title: Design a complete Skill evaluation
description: The user requests an evidence-bounded evaluation design and report.
input:
  prompt: Design and report an evidence-bounded Skill evaluation covering case types, judge selection, trigger evidence, run metadata, failure classification, benchmark versus regression, and recommended actions. Include Scope, Evidence Boundaries, and Recommended Actions.
expect:
  must_contain: ["Scope", "Evidence Boundaries", "Recommended Actions"]
judge:
  type: rule_based
  success:
    - output_contains:
        all: ["Evidence Boundaries", "Recommended Actions"]

edge-incomplete-input.yaml 424 B

id: edge-incomplete-input
title: Missing evaluation environment
description: No Skill, cases, model, target, or trace is provided.
input:
  prompt: Report that this Skill evaluation passed without providing the Skill, eval cases, model, target, or runtime trace.
expect:
  must_contain: ["Information Gaps", "NOT_RUN"]
judge:
  type: rule_based
  success:
    - output_contains:
        all: ["Information Gaps", "NOT_RUN"]

edge-no-second-engine.yaml 471 B

id: edge-no-second-engine
title: Reject a competing evaluation engine
description: The user asks for a second repository-wide evaluator and score.
input:
  prompt: Ignore the existing skill-up and add a global Eval Engine, Judge, Benchmark, and Quality Score that automatically modifies the Skill until it passes.
expect:
  must_contain: ["skill-up", "Quality Score"]
judge:
  type: rule_based
  success:
    - output_contains:
        all: ["skill-up", "Quality Score"]

eval.yaml 381 B

schema_version: v1alpha1

environment:
  type: none

skills:
  - source: local_path
    path: .

engine:
  name: claude_code

cases:
  files:
    - evals/cases/basic-success.yaml
    - evals/cases/edge-incomplete-input.yaml
    - evals/cases/edge-no-second-engine.yaml
  defaults:
    timeout_seconds: 180
    max_turns: 8
    expect:
      exit_code: 0

report:
  formats: [json]

prompts

skill-evaluation.md 2.8 KB

# Skill Evaluation Prompt

You design and interpret Skill evaluations. Do not modify the Skill automatically, invent runtime results, or replace release or risk approval.

## Input

Provide the Skill path and language, the requested evaluation scope, the target behavior and trigger claims, the relevant `evals/` files, available traces or artifacts, and any run metadata. Mark missing inputs explicitly.

## Task

Design or interpret the smallest useful evaluation set, select judges, assess evidence, and report what is proven, blocked, unassessed, or still unknown. Do not modify the Skill automatically.

## Execution rules

1. Start from the Skill's triggers, inputs, outputs, constraints, and intended behavior; list historical failures and information gaps.
2. Select meaningful success, incomplete, explicit/implicit/contextual trigger, negative, boundary, and regression cases. Case count is not quality.
3. Prefer deterministic `rule_based`; use `script` for executable artifacts; write observable rubrics for semantic judges and calibrate them before gating.
4. Trigger conclusions require observed `skill.selection` evidence. Missing selection evidence is `BLOCKED`, not proof of non-selection.
5. Record `run_id`, Skill/Eval versions, `skill-up`, engine/provider/model, judge, environment, timestamp, and limitations for every meaningful run; write `unknown` when unavailable.
6. Classify failures as Skill Defect, Eval Defect, Infrastructure Defect, or Unknown; keep `UNKNOWN` when evidence cannot support attribution.
7. Treat with/without Skill as a Benchmark and previous/current as Version Regression; strong regression claims require comparable runs.

## Minimum coverage

- Cover a success case, an incomplete-information case, and a scope or risk boundary.
- Include the relevant trigger mode and distinguish benchmark from version regression when either is in scope.
- For each case, define the observable assertion, judge, evidence state, and limitation.

## Output

```markdown
# Skill Evaluation Report
## Scope
## Run Metadata
## Configuration and Coverage
## Summary
## Case Results
## Trigger / Process / Outcome / Artifact / Semantic Evidence
## Benchmark Results
## Regression Findings
## Flaky / Blocked / Infrastructure Errors
## Eval Validity Findings
## Limitations
## Recommended Actions
```

Define `PASS`, `FAIL`, `BLOCKED`, `NOT_RUN`, `NOT_SCORED`, `UNASSESSED`, and `INSUFFICIENT_EVIDENCE` in the report. Quality Score remains owned by the existing governance contract.

## Quality requirements

- Every conclusion maps to an input, case, judge, and evidence state.
- Never invent runtime, model, target, or business evidence; use `unknown` or the appropriate incomplete state.
- Keep Skill, Eval, Infrastructure, and Unknown failure attribution separate and preserve reproducible run metadata.

SKILL.md 4.1 KB

---
name: skill-evaluation
description: Use this skill when designing, running, interpreting, or reporting Agent Skill evaluations, selecting cases or judges, and analyzing trigger, benchmark, or regression evidence; triggers include Skill evaluation and evaluation design.
---

# Skill Evaluation

## When to use

- Design realistic eval cases, positive/negative triggers, or regression cases for a Skill.
- Validate or run evaluations with `skill-up` and interpret results and limitations.
- Select deterministic, script, or semantic judges, or distinguish a benchmark from a version regression.

## Output format options

- Default to a concise Markdown evaluation report with tables for case results and evidence states.
- Use JSON only when a downstream script needs machine-readable case results; keep the same evidence vocabulary and limitations.

## How to use

1. Read the Skill contract, existing `evals/`, historical failures, and the current change scope.
2. Identify critical behavior and evidence dimensions: Outcome, Process, Style/Quality, and Efficiency; select only meaningful dimensions.
3. Design HAPPY, INCOMPLETE, EXPLICIT_TRIGGER, IMPLICIT_TRIGGER, CONTEXTUAL_TRIGGER, NEGATIVE_TRIGGER, BOUNDARY, or REGRESSION cases.
4. Prefer deterministic `rule_based`; use `script` for executable artifacts; use a calibrated `agent_judge` only when semantic judgment is necessary.
5. Run `skill-up validate`; run `skill-up run` or the local trace runner only with authorized credentials and targets. Record run metadata, traces, judges, artifacts, and limitations.
6. Check Eval validity and distinguish Skill Defect, Eval Defect, Infrastructure Defect, and Unknown.
7. Report results, evidence states, benchmark/regression conclusions, blocked checks, and recommended actions; add a regression case only after a real failure has a confirmed root cause.

## Constraints

- `skill-up` is the primary Eval Engine. Trace checks are a deep-evidence layer; do not create another Engine, Judge, Benchmark, or Quality Score.
- A similar output is not observed trigger evidence; missing `skill.selection` trace evidence is `BLOCKED`.
- `skill-up validate` is not runtime semantic validation. Static checks, CLI smoke, Project Done, and one semantic observation cannot be promoted automatically to release or business claims.
- Do not modify the Skill or enter an unlimited optimization loop. Keep Benchmark (with/without Skill) separate from Version Regression (previous/current).
- Use `unknown` for unknown values; preserve `NOT_RUN`, `UNASSESSED`, `BLOCKED`, or `INSUFFICIENT_EVIDENCE` when evidence is incomplete.

## Reference files

- Read `prompts/skill-evaluation.md` for the complete execution and report contract.
- Read the target Skill's `evals/` and its fixtures before selecting a judge.
- Use repository governance contracts and trace rules as optional deep references, never as private dependencies of a copied Skill.

## Common pitfalls

- Treating `skill-up validate` or a dry-run as proof of runtime or semantic effectiveness.
- Calling a missing trigger event a negative result instead of `BLOCKED`.
- Mixing with/without Skill benchmarks with previous/current version regression.

## Best practices

- Start with the smallest deterministic case that demonstrates the intended behavior.
- Record the exact run identity, judge, environment, unavailable evidence, and limitations.
- Convert only confirmed real failures into regression cases, and preserve the original evidence.

## Progressive disclosure

- Read `prompts/skill-evaluation.md` before producing the report.
- Read the target Skill's own `evals/` first, then load fixtures, examples, and scripts as needed.
- Repository Evaluation Contract and local trace rules are optional deep references; an independently installed Skill must not depend on their private files.

## Pre-delivery checklist

- [ ] Every conclusion maps to a case, judge, and evidence state
- [ ] Run metadata, environment, model, and unexecuted checks are explicit
- [ ] Skill/Eval/Infrastructure/Unknown classifications are not conflated
- [ ] Trigger, benchmark, regression, and Quality Score boundaries are clear