Claude Cursor GitHub Copilot Skill

improve-skill-quality

Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regression or underpowered, when a skill regressed after a change, when /evaluate repor

LLM Mart · 0 points · 20 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download dotnet-skills-.agents_skills_improve-skill-quality-d68dd70.zip · 12 KB

dotnet/skills 5471 418 forks MIT Updated 2d ago

Part of dotnet/skills — 119 skills

Install

skills CLI npx skills add https://github.com/dotnet/skills/tree/main/.agents/skills/improve-skill-quality

Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install dotnet-skills@llmmart

Git git clone https://github.com/dotnet/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole dotnet/skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Improve Skill Quality

Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake in this repo is rewriting skill prose in response to a verdict whose real cause was the eval, the fixtures, or the harness. Classify first, then fix.

When to Use

An evaluation verdict is a regression, underpowered, or "no credible improvement".
A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".
/evaluate reports "Evaluation ran but produced no results".
A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).
Deciding whether to strengthen or retire a persistently weak skill.

When Not to Use

Creating a new skill from scratch — use create-skill.
Creating a new eval.yaml from scratch — use create-skill-test.
Changing the harness itself (eng/skill-validator, eng/vally-adapter, evaluation*.yml).

Inputs

Input	Required	Description
Verdict evidence	Yes	The `/evaluate` PR comment, or `results.json` from the run artifacts
Losing trial transcripts	Yes for content fixes	Baseline vs. skilled output plus the judge's stated reason
Stimulus-vote W/T/L and repeated-run W/T/L	Yes	Separates cross-task evidence from reliability
Activation status per arm	Yes	Isolated and plugin activation are different failures

Workflow

Step 1: Get the evidence before forming a hypothesis

Read InvestigatingResults.md for how to download artifacts and read results.json. Extract, per failing stimulus:

authoritative stimulus-vote W/T/L and separate repeated-run W/T/L
activation status in the isolated and plugin arms, separately
the judge's verbatim reason on each losing trial
whether any trial errored, timed out, or produced empty output

Do not change skill content until you can quote a losing trial and the judge's reason for it. For the other cause classes the evidence is different: harness failures are diagnosed from the job log and the spec, and power problems from the trial record — neither has a losing trial to quote, and demanding one is what sends people rewriting prose instead.

Step 2: Classify the failure

Work down this table and stop at the first row that matches. Rows are ordered by how often the symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a fixture failure also presents as a setup or reliability failure and gets misfiled as one.

Symptom	Real cause class	Go to
A fixture does not build, is untracked by git, breaks for the wrong reason, or contradicts itself	Fixture	Step 4
No `results.json`, "produced no results", or the spec never loaded	Harness / spec-load	Step 3
Trials errored, timed out, or returned empty output	Reliability	Step 3
Trajectories unmatched, a trial errored, or the summary disagrees — verdict reported inconclusive	Reliability (not power)	Step 3
Positive record (e.g. 16W/8T/1L), comparison conclusive, verdict still not a pass	Statistical power	Step 5
Skilled arm equals baseline arm by construction	Eval design	Step 6
Activated and lost on quality, judge names a concrete defect	Skill content	Step 7
Activated in isolation, not in plugin	Activation / routing	Step 8
Not activated in either arm	Frontmatter description	Step 8
Wins but costs far more than baseline	Scope and cost	Step 7

A verdict is only a measured result when the comparison was conclusive: adapt.mjs requires zero errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or a regression. Confirm that before reading a record as a power problem.

Step 3: Rule out harness and reliability causes

See references/eval-triage.md for the full catalogue. The recurring ones:

A spec declaring both config: and defaults: is rejected by vally, the job still exits 0, and the PR comment blames "transient infrastructure". Merge them into one defaults: block.
An errored trial is not automatically a fixture problem — judge-side auth and session.idle failures look identical from the verdict and need harness fixes, not SDK pins.
expect_tools: [bash] on an advisory question forces a restore or build and turns an answer into a timeout with no quality gain.
Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails every grader and hides the real quality signal.
Unmatched trajectories, an errored trial, or a summary that disagrees make the comparison inconclusive: the remaining matched trials are biased, so the record is not a measured null and must not be read as a power or content problem.

Step 4: Verify the fixtures before touching the skill

Run python eng/eval-quality/check_eval_quality.py — it blocks eleven defect classes that can cost a real result here. Then confirm by hand:

every fixture behaves as its stimulus assumes — a fixture meant to be healthy builds, and one meant to be broken fails for the exact reason the stimulus is about and no other;
every referenced fixture is in the git index (git ls-files), not merely on disk — .gitignore has silently swallowed committed coverage fixtures;
a fixture never states the same fact in two places that disagree — a Cobertura report whose declared line-rate, summary totals and <line> elements differ is the canonical case — or the two arms legitimately read different truths.

Step 5: Check whether the eval could ever have passed

The gate has two independent bars, and confusing them is the usual misdiagnosis:

Distinct stimuli ≥ 5. Below that the verdict is reported underpowered — never a pass, never a regression.
The sign test must reach p ≤ 0.05 over the discordant (non-tie) stimulus votes. Ties are not discarded silently; they hold the discordant count down.

discordant stimulus votes	records that pass	p
≤ 4	none, however good the skill	≥ 0.0625
5–7	zero losses only (5W/0L)	0.031
8	one loss survivable (7W/1L)	0.035

So at exactly 5 stimuli a single tie is fatal — it leaves 4 discordant. At 6 stimuli one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.

So a positive record with a failing verdict is a power problem, not a content problem. Fix it by adding discriminating stimuli. Raising runs measures reliability for the same task and cannot clear the floor.

Step 6: Check whether the two arms differ at all

An eval that compares the skill against itself measures judge noise:

A dormancy guard (expect_activation: false) must not also set constraints.reject_skills. That makes the skilled arm skill-free, so the activation contract cannot observe a hijack. Schema version 4 retains the identical-arm comparison for diagnostics but excludes it from preference inference; unexpected isolated activation still blocks a pass.
A skill with disable-model-invocation: true is absent from the model-facing skilled arm, so its direct eval compares two identical arms regardless of whether graders inspect activation or answer content. Cover it through consumer outcomes instead; for example, filter-syntax is covered by run-tests and mtp-hot-reload.
A grader whose config is missing its required key enforces nothing, so the stimulus has one fewer assertion than it appears to.

Step 7: Fix skill content against the losing trial

Only now change the skill. Apply the patterns in references/writing-for-baseline-delta.md; the ones that most often flip a loss:

Replace reference prose the model already knows with decisions it would otherwise get wrong.
Add stop-conditions so a strong skill does not over-apply — but do not over-correct into answering more narrowly than the baseline did.
Scale output structure to input size; a dashboard for an 8-test suite loses to a direct answer.
Require truthful validation reporting; claiming "Build succeeded" after a failed restore is an automatic loss.
Verify load-bearing API claims by compiling or probing, not by reading source.
For cost regressions, gate rare or expensive paths behind references/ reads and size any orchestration to the user's scope.

Step 8: Fix activation

Activation failures are frontmatter and routing failures, not body failures. See references/eval-triage.md. Summary:

Failure	Fix
Not activated in any arm	Put the user's own words in `description`: symptoms, error codes, artifact names, quoted requests
A sibling skill wins the prompt	Claim the exact ambiguous words in `description`, and add matching exclusions on both siblings
Model answers with no skill at all	Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm
Boundary excludes real scenarios	Re-read every "do not use for" clause against every eval prompt and real workflow phase
Description at the 1,024-char ceiling	Cut restated body content, not trigger phrases; check the plugin menu budget too

Step 9: Re-validate

dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
python eng/eval-quality/check_eval_quality.py
./eng/run-skill-evals.sh <plugin> <skill>

Then request the official run by submitting a PR review containing /evaluate (Files changed → Review changes), which binds the run to the reviewed commit. Before declaring a regression on the result, confirm the skill payload actually changed — reruns on byte-identical content have shifted 7W/2T/2L to 4W/5T/2L.

Validation

For a content fix, a losing trial and the judge's stated reason are quoted in the PR description.
The failure was classified before any content was edited.
check_eval_quality.py and skill-validator check both pass.
Distinct-stimulus count clears the power bar for the target effect and observed tie rate.
Isolated and plugin activation are both reported.
The PR body records root cause, fix, and validation so the lesson is reusable.

Common Pitfalls

Pitfall	Solution
Rewriting skill prose in response to an underpowered verdict	Underpowered means too few distinct stimuli; add discriminating stimuli instead
Adding `defaults: runs:` to a spec that already has `config:`	Merge into a single `defaults:` block; vally rejects specs with both
Padding `runs` to clear the stimulus floor	Repeats measure reliability for one task; add stimuli
Treating an errored trial as fixture nondeterminism	Read the stderr first; judge-side auth failures need harness fixes
Fixing a "wrong" answer that the fixture actually made wrong	Check fixture self-consistency before blaming the response
Strengthening a skill nobody uses and nothing passes	Weak eval signal plus thin telemetry is a valid retirement case
Landing a fix without re-running	Verify the invoked payload contains the fix; judge noise is real

References

references/writing-for-baseline-delta.md — content patterns that beat the unskilled model
references/eval-triage.md — symptom, cause and fix catalogue with PR citations
eng/eval-quality/README.md — the eleven structural gate checks and why each exists
eng/vally-adapter/InvestigatingResults.md — downloading artifacts and reading results.json. This is the current guide; the similarly-named eng/skill-validator/src/docs/InvestigatingResults.md documents the retired skill-validator evaluate schema and does not describe today's results.

Files (skills)

references

eval-triage.md 8.6 KB

# Evaluation triage catalogue

Symptom → cause → fix, with the PR where each was diagnosed. Use with
`SKILL.md` Step 2; this file is the detail behind each cause class.

## Harness and reliability

| Symptom | Cause | Fix | Evidence |
|---------|-------|-----|----------|
| "Evaluation ran but produced no results", advice says transient infrastructure | Spec declares both `config:` and `defaults:`; vally throws, the job still exits 0 | Merge into one `defaults:` block carrying `timeout` and `runs` | PR #971 |
| Same message, nothing in `plugins/` changed | Genuine LLM-session auth failure | Re-post `/evaluate`; inspect job logs before touching content | PR #932 |
| Trial errored, avgN unusually low | Judge-side CAPI / `session.idle` timeout, not fixture nondeterminism | Read the trial stderr first; fix the harness, do not pin an SDK | PR #907 |
| Timeout on an advisory question | `expect_tools: [bash]` forced a restore or build | Drop the tool requirement; the answer was always textual | PR #861 |
| Every grader fails and output is empty | Code-generation stimulus timed out | Raise to ~360s | PR #862, PR #863 |
| Only the skilled arm aborts with "contains no SKILL.md" | A setup cleanup command deleted the staged skill directory | Skip directories carrying `SKILL.md` when stripping sources | PR #878 |
| Trials silently dropped | Setup command exited non-zero although its artifact was produced | Guard intentional failures, e.g. `dotnet build -bl \|\| exit 0` | PR #878 |
| One arm has unmatched trajectories | Comparison judge error or arm timeout biases the remaining trials | Report inconclusive, do not read it as a regression | PR #887 |
| Fixture resolves locally, SDK-not-found in CI | Fixture pin without roll-forward | `global.json` with `rollForward: latestMajor` | PR #907 |

## Fixtures

| Symptom | Cause | Fix | Evidence |
|---------|-------|-----|----------|
| Judge penalizes the agent for "pre-existing build issues" | A fixture meant to be healthy does not compile | Build every healthy fixture before shipping the eval; a deliberately broken one must fail only for the reason its stimulus is about | PR #949 |
| Scenario fails at setup in CI but passes locally | Fixture is on disk but not in the git index (`.gitignore` swallowed it) | Verify with `git ls-files`; the gate now blocks this | PR #945, PR #953 |
| Judge says the response "made a critical error" about a value the fixture supplies | The fixture states the same fact in two places and they disagree, so each arm can legitimately read a different truth — e.g. a Cobertura report whose declared `line-rate` contradicts its `<line>` payload or summary totals | Make every representation of the value agree, then re-derive any rubric or prompt that quotes it | PR #964, PR #945 |
| Baseline scores suspiciously well | The fixture never reproduces the bug the stimulus is named for | Rebuild the fixture until it produces the real error | PR #974 |
| `n` rose but power did not | Duplicate or rename-leftover fixtures wired in as new stimuli | Delete byte-equivalent leftovers; only wire fixtures exercising new behavior | PR #971, PR #945 |

## Statistical power

The gate has two independent bars: **distinct stimuli ≥ 5** (else `underpowered`), and **p ≤ 0.05
on an exact one-sided sign test over discordant (non-tie) stimulus votes**. Repeated runs collapse
to one vote per stimulus and remain reliability evidence.

| discordant stimulus votes | records that pass | p |
|---:|---|---:|
| ≤ 4 | none | ≥ 0.0625 |
| 5–7 | zero losses only (5W/0L) | 0.031 |
| 8 | one loss survivable (7W/1L) | 0.035 |

At exactly 5 stimuli one tie is fatal — it leaves 4 discordant votes. At 6 stimuli one tie is
survivable; at 7, up to two are. A loss is not: 4W/3T/1L over eight stimulus votes fails.

Consequences seen in real runs:

- Five `dotnet-test` evals raised to exactly 5 distinct stimuli returned 16W/8T/1L overall — every skill
  winning, none regressing — and **all five failed**, four because ties made a pass unreachable
  before the run started. (PR #971, `eng/eval-quality/README.md`)
- At the 32% tie rate measured there, a genuinely-helping skill parked at 5 stimulus votes is
  certified about one run in ten; at 15 stimulus votes, about nine in ten.
- Adding stimuli increases task breadth. Raising `runs` measures reliability for the same tasks and
  cannot clear the stimulus floor.
- The verdict reads each repeated run's **winner**, never its magnitude: weighting a confidence interval by
  "slightly better" vs "much better" made a stronger win look like variance and reversed verdicts on
  identical records. (PR #965, PR #952)

## Eval design

| Symptom | Cause | Fix | Evidence |
|---------|-------|-----|----------|
| A dormancy guard scores randomly across runs | Legacy results counted an identical-arm dormancy comparison in preference; `constraints.reject_skills` also prevents observing the activation contract | On schema version 4, use `expect_activation: false` alone; inspect `excludedScenarioEvidence` and `activationContract` instead of preference | PR #945, PR #953 |
| A reference skill shows no improvement | `disable-model-invocation: true` keeps it out of the model-facing skilled arm, so any direct eval compares identical arms | Remove the direct eval and cover the reference through consumer outcomes | PR #971, PR #976, issue #899 |
| An eval "passes" while the skill stopped emitting its signature output | No grader asserts the mandated shape | Add a grader for the exact contract (e.g. the `Recommendation:` line) | PR #904 |
| Overfit score high, user value unclear | Rubric items reward using the skill, or prompts echo skill vocabulary | Drop them: the harness already reports activation separately, so a rubric never needs to. Keep rubric items outcome-shaped and de-cue the prompt | PR #904 |
| Both arms produce the same kind of artifact and the judge falls back on comparing volume | The rubric rewards raw output instead of the property under test | Add anti-hijack criteria: do not invoke the skill, and do not reward quantity (number of tests, findings, or lines produced) | PR #945 |
| A grader appears to enforce something but does not | `config:` is missing its required key after an indentation slip | `check_eval_quality.py` blocks it; verify the key is present | `eng/eval-quality/README.md` |
| Two stimuli behave identically | Duplicate YAML key — a leftover `prompt:`/`graders:` block overwrites the following stimulus field by field | Delete the stray block after confirming it is not a distinct stimulus that lost its `- name:` | PR #971 |
| Eval measures path recall | The skill is a map to reference files | Do not create the eval; test the consumer's outcome instead | PR #974 |

## Activation

| Symptom | Cause | Fix | Evidence |
|---------|-------|-----|----------|
| Not activated in any arm | Triggering words absent from `description` | Add symptoms, error codes, artifact names in user language | PR #974 |
| Not activated for the scenarios the skill exists for | An over-broad exclusion clause | Re-read every "do not use for" clause against every eval prompt | PR #974 |
| Wrong sibling wins the prompt | Descriptions partitioned by topic instead of by discriminator | Partition on the real question, and add handoff exclusions on both sides | PR #864 |
| Sibling wins on one ambiguous word | The target skill never claims that word | Claim it explicitly — "review" had to be claimed by `writing-mstest-tests` | PR #863 |
| Isolated activation perfect, plugin arm fails | The model self-serves: reads the file and answers with no skill at all | Raise stakes in the description, de-crowd the menu, verify in the plugin arm | PR #850 |
| Menu pressure across a plugin | Helper/reference skills consuming budget | `disable-model-invocation: true` keeps them invocable by name only | PR #850 |

## Process

| Rule | Evidence |
|------|----------|
| Trigger `/evaluate` by submitting a PR review (Files changed → Review changes) so the run binds to the reviewed commit | PR #956, PR #949 |
| Before declaring a regression, confirm the invoked payload changed — reruns on byte-identical content moved 7W/2T/2L to 4W/5T/2L | PR #974 |
| Use cross-family evaluation for broad rewrites; single-family passes hide model-specific regressions | issue #899, PR #947 |
| Workflow changes cannot be validated by the PR's own evaluation (GitHub runs workflow definitions from `main`) — use manual dispatches | PR #872 |
| Prefer deterministic scripts over agentic workflows for deterministic policy | PR #928 |
| Agent `tools:` allowlists are host-specific and case-sensitive; an allowlist can grant zero tools | PR #856, PR #847 |
| Keep `InvestigatingResults.md` in sync whenever verdict fields, scoring, or PR-comment wording change | PR #965, PR #932 |

writing-for-baseline-delta.md 7.2 KB

# Writing skill content that beats the baseline

Every skill in this repo is scored head-to-head against the *same model with no skill loaded*.
The score is a **delta**, so content the model already produces unaided is worth zero — and
content that makes the model slower, longer, or more hedged is worth less than zero.

Each rule below is traced to the PR where it was learned.

## 1. Encode decisions, not knowledge

**Rule:** Write what the model should *do* when it sees a symptom, not what an API *is*.

`system-text-json-net11` scored 0% improvement because it "read as reference prose — API
signatures the model already reproduces". The rewrite into imperative decision guidance is what
moved it. (PR #926)

Practical test: delete any sentence the unskilled model would have written anyway. If most of the
skill disappears, it is a reference doc, not a skill.

## 2. Use "when A, do B, never C, verify D" tables

**Rule:** Route with decision tables so the model picks one answer instead of listing plausible
alternatives.

PR #926 added a table mapping PascalCase / typed-metadata / probing requests to exact APIs and
their anti-patterns. PR #947 added "Rules That Change the Answer" tables across the MAUI skills.
Tables also give graders something concrete to assert on.

## 3. Demand a concrete final artifact

**Rule:** Specify the exact shape of the answer — the command, the verdict line, the findings
table, the recommendation.

Generic advice did not move the template-engine skills; output contracts did: "exact single
`dotnet new` command", "one-line verdict header", "single findings table", decisive
`Recommendation:` lines. (PR #904)

If the skill mandates an output shape, the eval must assert on that shape, or the skill can
silently stop emitting it.

## 4. Scale structure to input size

**Rule:** Preserve the full report template for large inputs; answer directly for small ones.

`assertion-quality` regressed on an 8-test fixture because the skill forced a "Summary Dashboard +
12 categories" onto a trivial input and lost to a concise baseline. The fix instructed the skill to
scale report depth to suite size and complexity. (PR #865)

## 5. Add stop-conditions — then check you have not over-corrected

**Rule:** Say when *not* to act; then verify the skilled answer is still at least as complete as
the baseline's.

Strong skills regress by doing too much: rewriting working code, answering beyond the question,
prescribing a remedy before measuring. PR #910 added stop-conditions to `eval-performance` so it
would not act on compile-time slowness or unmeasured builds; PR #947 added scope control to
`maui-collectionview`.

The opposite failure is just as real: in PR #947 the scope-control wording "over-corrected the
other skills into answering too narrowly", and failing scenarios had skilled answers *shorter* than
baseline. Length is not the goal, but omitting the implementation detail the baseline supplied is a
loss.

## 6. Do not make discoverable inputs "required"

**Rule:** Only mark an input required when a human genuinely must supply it. Otherwise instruct the
agent to discover it.

A `Project or solution path | Yes` row in an Inputs table made the agent answer "I need to see your
project file" while `TestProject.csproj` sat in the working directory. (PR #974)

## 7. Verify load-bearing claims empirically

**Rule:** Compile or probe anything the skill asserts about API surface or runtime behavior.

The most damaging skill defects found in review were factual: MAUI docs taught `ItemSizingStrategy`
on `LinearItemsLayout`, which does not compile (MAUIX2002), and a theming "fix" was reverted after a
runtime probe disproved the source-reading that motivated it. (PR #947)

## 8. Preserve semantics in migration mappings

**Rule:** Migration tables need semantic guardrails, not just API substitutions.

A one-cell mapping steered frontier models into a behavior change: `TimeProvider` guidance using
`.DateTime` silently set `DateTimeKind.Unspecified`; the correct mapping is `.UtcDateTime` /
`.LocalDateTime`. (PR #906)

## 9. Require truthful validation reporting

**Rule:** Tell the agent to distinguish restore, build and test failures, and to cite a clean run
before claiming success.

`migrate-static-to-wrapper` lost trials for claiming "Build succeeded" after a restore failure;
`code-testing-agent` had to be told to cite a clean run. (PR #945)

## 10. Prove already-correct inputs are left alone

**Rule:** Any skill that migrates or rewrites code must state the no-op condition, and the eval must
test it.

PR #929 added an "already on v3" boundary fixture verifying no changes were made; the reviewer
called it "the single best addition" in the PR.

## 11. Keep the common path in `SKILL.md`, gate the rest

**Rule:** Rare, expensive or platform-specific paths belong behind a `references/` read.

`coverage-analysis` went from 30 KB to 15 KB by moving PowerShell and report-generation paths into
references read only when needed — cost down, contract unchanged. (PR #971)

## 12. Size orchestration to the request

**Rule:** Do not run a full research → plan → implement pipeline for one function.

`code-testing-agent` was split into focused and broad paths so a single-function request skips
`.testagent/` artifacts and extra passes. (PR #971)

## 13. Structure beats verbosity

**Rule:** When a scenario ties despite a longer skilled answer, the missing differentiator is a
recommended shape, not more words.

Both arms hardcoded colors in the MAUI theming scenario. The winning change was the rule "define
the palette once — don't scatter literals", not a longer explanation. (PR #947)

## 14. Retirement is a legitimate outcome

**Rule:** A skill that is weak across model families, thinly used, and costing menu budget should be
cut, not polished indefinitely.

`mcp-csharp-debug` was cut after strengthening 0 of 5 families, with owner confirmation; the change
removed the skill, its eval, its CODEOWNERS entry and cross-skill references. (PR #938)
`dotnet-test-frameworks` was removed as a duplicate subset that "nothing actually loaded". (PR #851)

## Frontmatter: the description is the router

The `description` is the only text the runtime sees when deciding whether to load the skill. (PR #974)

| Rule | Evidence |
|------|----------|
| Include symptoms, error codes and artifact names in the user's words (`CS1501`, `.testsettings`, `MSTEST0014`) | PR #974 |
| Lead with an action verb and quote natural requests ("what's wrong with my build file?") | PR #910 |
| Partition siblings by the real discriminator ("abstraction already exists" vs "create wrapper first"), with exclusions on both sides | PR #864 |
| Claim the ambiguous words that route to the wrong sibling — `writing-mstest-tests` had to claim "review" | PR #863 |
| Never exclude a phase the skill exists to serve; "already on MSTest v3+" blocked its own post-bump fixtures | PR #974 |
| Watch both limits: 1,024 characters per description **and** the plugin menu budget | PR #974, PR #910 |
| A helper skill that users should not invoke can set `disable-model-invocation: true` to free menu budget | PR #850 |

Menu pressure is measurable: disabling model invocation for one helper skill dropped the
`dotnet-test` menu from 14,981 to 14,261 characters (PR #850), and PR #910 tracked the msbuild menu
budget explicitly as part of a description rewrite.

SKILL.md 12.4 KB

---
name: improve-skill-quality
description: Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regression or underpowered, when a skill regressed after a change, when /evaluate reports no results, or when deciding whether a weak skill should be strengthened or retired. Do not use for scaffolding a brand-new skill (use create-skill) or a brand-new eval (use create-skill-test).
---

# Improve Skill Quality

Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake
in this repo is rewriting skill prose in response to a verdict whose real cause was the eval,
the fixtures, or the harness. Classify first, then fix.

## When to Use

- An evaluation verdict is a regression, underpowered, or "no credible improvement".
- A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".
- `/evaluate` reports "Evaluation ran but produced no results".
- A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).
- Deciding whether to strengthen or retire a persistently weak skill.

## When Not to Use

- Creating a new skill from scratch — use `create-skill`.
- Creating a new `eval.yaml` from scratch — use `create-skill-test`.
- Changing the harness itself (`eng/skill-validator`, `eng/vally-adapter`, `evaluation*.yml`).

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| Verdict evidence | Yes | The `/evaluate` PR comment, or `results.json` from the run artifacts |
| Losing trial transcripts | Yes for content fixes | Baseline vs. skilled output plus the judge's stated reason |
| Stimulus-vote W/T/L and repeated-run W/T/L | Yes | Separates cross-task evidence from reliability |
| Activation status per arm | Yes | Isolated and plugin activation are different failures |

## Workflow

### Step 1: Get the evidence before forming a hypothesis

Read [InvestigatingResults.md](../../../eng/vally-adapter/InvestigatingResults.md) for how to
download artifacts and read `results.json`. Extract, per failing stimulus:

- authoritative stimulus-vote W/T/L and separate repeated-run W/T/L
- activation status in the **isolated** and **plugin** arms, separately
- the judge's verbatim reason on each losing trial
- whether any trial errored, timed out, or produced empty output

Do not change skill content until you can quote a losing trial and the judge's reason for it. For the
other cause classes the evidence is different: harness failures are diagnosed from the job log and
the spec, and power problems from the trial record — neither has a losing trial to quote, and
demanding one is what sends people rewriting prose instead.

### Step 2: Classify the failure

Work down this table and stop at the first row that matches. Rows are ordered by how often the
symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a
fixture failure also presents as a setup or reliability failure and gets misfiled as one.

| Symptom | Real cause class | Go to |
|---------|------------------|-------|
| A fixture does not build, is untracked by git, breaks for the wrong reason, or contradicts itself | Fixture | Step 4 |
| No `results.json`, "produced no results", or the spec never loaded | Harness / spec-load | Step 3 |
| Trials errored, timed out, or returned empty output | Reliability | Step 3 |
| Trajectories unmatched, a trial errored, or the summary disagrees — verdict reported inconclusive | Reliability (not power) | Step 3 |
| Positive record (e.g. 16W/8T/1L), comparison conclusive, verdict still not a pass | Statistical power | Step 5 |
| Skilled arm equals baseline arm by construction | Eval design | Step 6 |
| Activated and lost on quality, judge names a concrete defect | Skill content | Step 7 |
| Activated in isolation, not in plugin | Activation / routing | Step 8 |
| Not activated in either arm | Frontmatter description | Step 8 |
| Wins but costs far more than baseline | Scope and cost | Step 7 |

A verdict is only a *measured* result when the comparison was conclusive: `adapt.mjs` requires zero
errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or
a regression. Confirm that before reading a record as a power problem.

### Step 3: Rule out harness and reliability causes

See [references/eval-triage.md](references/eval-triage.md) for the full catalogue. The recurring ones:

- A spec declaring both `config:` and `defaults:` is rejected by vally, the job still exits 0, and
  the PR comment blames "transient infrastructure". Merge them into one `defaults:` block.
- An errored trial is not automatically a fixture problem — judge-side auth and `session.idle`
  failures look identical from the verdict and need harness fixes, not SDK pins.
- `expect_tools: [bash]` on an advisory question forces a restore or build and turns an answer into
  a timeout with no quality gain.
- Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails
  every grader and hides the real quality signal.
- Unmatched trajectories, an errored trial, or a summary that disagrees make the comparison
  **inconclusive**: the remaining matched trials are biased, so the record is not a measured null
  and must not be read as a power or content problem.

### Step 4: Verify the fixtures before touching the skill

Run `python eng/eval-quality/check_eval_quality.py` — it blocks eleven defect classes that can
cost a real result here. Then confirm by hand:

- every fixture behaves as its stimulus assumes — a fixture meant to be healthy builds, and one
  meant to be broken fails for the exact reason the stimulus is about and no other;
- every referenced fixture is in the git index (`git ls-files`), not merely on disk — `.gitignore`
  has silently swallowed committed coverage fixtures;
- a fixture never states the same fact in two places that disagree — a Cobertura report whose
  declared `line-rate`, summary totals and `<line>` elements differ is the canonical case — or the
  two arms legitimately read different truths.

### Step 5: Check whether the eval could ever have passed

The gate has two independent bars, and confusing them is the usual misdiagnosis:

1. **Distinct stimuli ≥ 5.** Below that the verdict is reported
   `underpowered` — never a pass, never a regression.
2. **The sign test must reach p ≤ 0.05 over the *discordant* (non-tie) stimulus votes.** Ties are not
   discarded silently; they hold the discordant count down.

| discordant stimulus votes | records that pass | p |
|---:|---|---:|
| ≤ 4 | none, however good the skill | ≥ 0.0625 |
| 5–7 | zero losses only (5W/0L) | 0.031 |
| 8 | one loss survivable (7W/1L) | 0.035 |

So at exactly 5 stimuli a single tie is fatal — it leaves 4 discordant. At 6 stimuli
one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.

So a positive record with a failing verdict is a power problem, not a content problem. Fix it by
adding **discriminating stimuli**. Raising `runs` measures reliability for the same task and cannot
clear the floor.

### Step 6: Check whether the two arms differ at all

An eval that compares the skill against itself measures judge noise:

- A dormancy guard (`expect_activation: false`) must **not** also set `constraints.reject_skills`.
  That makes the skilled arm skill-free, so the activation contract cannot observe a hijack.
  Schema version 4 retains the identical-arm comparison for diagnostics but excludes it from
  preference inference; unexpected isolated activation still blocks a pass.
- A skill with `disable-model-invocation: true` is absent from the model-facing skilled arm, so its
  direct eval compares two identical arms regardless of whether graders inspect activation or answer
  content. Cover it through consumer outcomes instead; for example, `filter-syntax` is covered by
  `run-tests` and `mtp-hot-reload`.
- A grader whose `config` is missing its required key enforces nothing, so the stimulus has one
  fewer assertion than it appears to.

### Step 7: Fix skill content against the losing trial

Only now change the skill. Apply the patterns in
[references/writing-for-baseline-delta.md](references/writing-for-baseline-delta.md); the ones that
most often flip a loss:

- Replace reference prose the model already knows with decisions it would otherwise get wrong.
- Add stop-conditions so a strong skill does not over-apply — but do not over-correct into
  answering more narrowly than the baseline did.
- Scale output structure to input size; a dashboard for an 8-test suite loses to a direct answer.
- Require truthful validation reporting; claiming "Build succeeded" after a failed restore is an
  automatic loss.
- Verify load-bearing API claims by compiling or probing, not by reading source.
- For cost regressions, gate rare or expensive paths behind `references/` reads and size any
  orchestration to the user's scope.

### Step 8: Fix activation

Activation failures are frontmatter and routing failures, not body failures. See
[references/eval-triage.md](references/eval-triage.md). Summary:

| Failure | Fix |
|---------|-----|
| Not activated in any arm | Put the user's own words in `description`: symptoms, error codes, artifact names, quoted requests |
| A sibling skill wins the prompt | Claim the exact ambiguous words in `description`, and add matching exclusions on **both** siblings |
| Model answers with no skill at all | Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm |
| Boundary excludes real scenarios | Re-read every "do not use for" clause against every eval prompt and real workflow phase |
| Description at the 1,024-char ceiling | Cut restated body content, not trigger phrases; check the plugin menu budget too |

### Step 9: Re-validate

```bash
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
python eng/eval-quality/check_eval_quality.py
./eng/run-skill-evals.sh <plugin> <skill>
```

Then request the official run by submitting a PR review containing `/evaluate` (Files changed →
Review changes), which binds the run to the reviewed commit. Before declaring a regression on the
result, confirm the skill payload actually changed — reruns on byte-identical content have shifted
7W/2T/2L to 4W/5T/2L.

## Validation

- [ ] For a content fix, a losing trial and the judge's stated reason are quoted in the PR description.
- [ ] The failure was classified before any content was edited.
- [ ] `check_eval_quality.py` and `skill-validator check` both pass.
- [ ] Distinct-stimulus count clears the power bar for the target effect and observed tie rate.
- [ ] Isolated **and** plugin activation are both reported.
- [ ] The PR body records root cause, fix, and validation so the lesson is reusable.

## Common Pitfalls

| Pitfall | Solution |
|---------|----------|
| Rewriting skill prose in response to an underpowered verdict | Underpowered means too few distinct stimuli; add discriminating stimuli instead |
| Adding `defaults: runs:` to a spec that already has `config:` | Merge into a single `defaults:` block; vally rejects specs with both |
| Padding `runs` to clear the stimulus floor | Repeats measure reliability for one task; add stimuli |
| Treating an errored trial as fixture nondeterminism | Read the stderr first; judge-side auth failures need harness fixes |
| Fixing a "wrong" answer that the fixture actually made wrong | Check fixture self-consistency before blaming the response |
| Strengthening a skill nobody uses and nothing passes | Weak eval signal plus thin telemetry is a valid retirement case |
| Landing a fix without re-running | Verify the invoked payload contains the fix; judge noise is real |

## References

- [references/writing-for-baseline-delta.md](references/writing-for-baseline-delta.md) — content patterns that beat the unskilled model
- [references/eval-triage.md](references/eval-triage.md) — symptom, cause and fix catalogue with PR citations
- [eng/eval-quality/README.md](../../../eng/eval-quality/README.md) — the eleven structural gate checks and why each exists
- [eng/vally-adapter/InvestigatingResults.md](../../../eng/vally-adapter/InvestigatingResults.md) — downloading artifacts and reading `results.json`. This is the current guide; the similarly-named `eng/skill-validator/src/docs/InvestigatingResults.md` documents the retired `skill-validator evaluate` schema and does not describe today's results.