This document is the authoritative reference for how PluginEval measures plugin and skill quality.\nIt covers the three evaluation layers, all ten scoring dimensions, the composite formula, badge\nthresholds, anti-pattern flags, Elo ranking, and actionable improvement tips.
\nRelated: Full rubric anchors
\nPluginEval stacks three complementary layers. Each layer produces a score between 0.0 and 1.0 for\neach applicable dimension, and later layers override or blend with earlier ones according to\nper-dimension blend weights.
\nSpeed: < 2 seconds. No LLM calls. Deterministic.
\nThe static analyzer (layers/static.py) runs six sub-checks directly against the parsed SKILL.md:
| Sub-check | \nWhat it measures | \n
|---|---|
frontmatter_quality | \nName presence, description length, trigger-phrase quality | \n
orchestration_wiring | \nOutput/input documentation, code block count, orchestrator anti-pattern | \n
progressive_disclosure | \nLine count vs. sweet-spot (200–600 lines), references/ and assets/ bonuses | \n
structural_completeness | \nHeading density, code blocks, examples section, troubleshooting section | \n
token_efficiency | \nMUST/NEVER/ALWAYS density, duplicate-line repetition ratio | \n
ecosystem_coherence | \nCross-references to other skills/agents, \"related\"/\"see also\" mentions | \n
These six sub-checks feed directly into six of the ten final dimensions (via STATIC_TO_DIMENSION\nmapping). The remaining four dimensions — output_quality, scope_calibration,\nrobustness, and part of triggering_accuracy — receive no static contribution and rely\nentirely on Layer 2 and/or Layer 3.
Anti-pattern penalty is applied multiplicatively to the Layer 1 score:
\npenalty = max(0.5, 1.0 − 0.05 × anti_pattern_count)\nEach additional detected anti-pattern reduces the score by 5%, flooring at 50%.
\nSpeed: 30–90 seconds. One or more LLM calls (Sonnet by default). Non-deterministic.
\nThe eval-judge agent reads the SKILL.md and any references/ files, then scores four\ndimensions using anchored rubrics (see references/rubrics.md):
The judge returns a structured JSON object (no markdown fences) that the eval engine merges\ninto the composite. When judges > 1, scores are averaged and Cohen's kappa is reported as\nan inter-judge agreement metric.
Speed: 5–20 minutes. N=50 simulated Agent SDK invocations (default). Statistical.
\nMonte Carlo runs N real prompts through the skill and records:
The Layer 3 composite formula:
\nmc_score = 0.40 × activation_rate\n + 0.30 × (1 − min(1.0, CV))\n + 0.20 × (1 − failure_rate)\n + 0.10 × efficiency_norm\nwhere efficiency_norm = max(0, 1 − median_tokens / 8000).
The final score is a weighted blend across all three layers for each dimension, then summed:
\ncomposite = Σ(dimension_weight × blended_dimension_score) × 100 × anti_pattern_penalty\n| Dimension | \nWeight | \nWhy it matters | \n
|---|---|---|
triggering_accuracy | \n0.25 | \nA skill that never fires — or fires incorrectly — has no value | \n
orchestration_fitness | \n0.20 | \nSkills must be pure workers; supervisor logic belongs in agents | \n
output_quality | \n0.15 | \nCorrect, complete output is the primary deliverable | \n
scope_calibration | \n0.12 | \nNeither a stub nor a bloated monster | \n
progressive_disclosure | \n0.10 | \nSKILL.md is lean; detail lives in references/ | \n
token_efficiency | \n0.06 | \nMinimal context waste per invocation | \n
robustness | \n0.05 | \nHandles edge cases without crashing | \n
structural_completeness | \n0.03 | \nCorrect sections in the right order | \n
code_template_quality | \n0.02 | \nWorking, copy-paste-ready examples | \n
ecosystem_coherence | \n0.02 | \nCross-references; no duplication with siblings | \n
Each dimension draws from different layers at different ratios. With all three layers active\n(--depth deep or certify):
| Dimension | \nStatic | \nJudge | \nMonte Carlo | \n
|---|---|---|---|
triggering_accuracy | \n0.15 | \n0.25 | \n0.60 | \n
orchestration_fitness | \n0.10 | \n0.70 | \n0.20 | \n
output_quality | \n0.00 | \n0.40 | \n0.60 | \n
scope_calibration | \n0.30 | \n0.55 | \n0.15 | \n
progressive_disclosure | \n0.80 | \n0.20 | \n0.00 | \n
token_efficiency | \n0.40 | \n0.10 | \n0.50 | \n
robustness | \n0.00 | \n0.20 | \n0.80 | \n
structural_completeness | \n0.90 | \n0.10 | \n0.00 | \n
code_template_quality | \n0.30 | \n0.70 | \n0.00 | \n
ecosystem_coherence | \n0.85 | \n0.15 | \n0.00 | \n
At --depth standard (static + judge only), blends are renormalized to drop the Monte Carlo\ncolumn. At --depth quick (static only), all weight falls on Layer 1.
For a given depth, the blended score for dimension d is:
blended[d] = Σ( layer_weight[d][layer] × layer_score[d][layer] )\n ─────────────────────────────────────────────────────\n Σ( layer_weight[d][layer] for available layers )\nThis normalization ensures that skipping Monte Carlo at standard depth doesn't artificially\ndeflate scores.
\nEach dimension score is a float in [0.0, 1.0]. The CLI converts it to a letter grade:
| Grade | \nScore range | \nMeaning | \n
|---|---|---|
| A | \n0.90 – 1.00 | \nExcellent — no meaningful improvement needed | \n
| B | \n0.80 – 0.89 | \nGood — minor gaps only | \n
| C | \n0.70 – 0.79 | \nAdequate — one or two clear improvement areas | \n
| D | \n0.60 – 0.69 | \nMarginal — needs targeted work | \n
| F | \n< 0.60 | \nFailing — significant remediation required | \n
When reading a report, focus first on the lowest-graded dimension that has the highest weight.\nA D in triggering_accuracy (weight 0.25) costs far more than a D in ecosystem_coherence\n(weight 0.02).
Confidence intervals appear in the report when Layer 2 or Layer 3 ran. Narrow CIs (± < 5\npoints) indicate stable scores. Wide CIs suggest inconsistency — often caused by an ambiguous\ndescription or instructions that work for some prompt styles but not others.
\nBadges require both a composite score threshold AND an Elo threshold (when Elo is available).\nThe Badge.from_scores() logic checks composite first, then Elo if provided:
| Badge | \nComposite | \nElo | \nMeaning | \n
|---|---|---|---|
| Platinum ★★★★★ | \n≥ 90 | \n≥ 1600 | \nReference quality — suitable for gold corpus | \n
| Gold ★★★★ | \n≥ 80 | \n≥ 1500 | \nProduction ready | \n
| Silver ★★★ | \n≥ 70 | \n≥ 1400 | \nFunctional, has improvement opportunities | \n
| Bronze ★★ | \n≥ 60 | \n≥ 1300 | \nMinimum viable — not yet recommended for users | \n
| — | \n< 60 | \nany | \nDoes not meet minimum bar | \n
The Elo threshold is skipped when Elo has not been computed (i.e., at quick or standard depth\nwithout certify). A skill can earn a badge on composite score alone in those cases.
The static analyzer detects five anti-patterns. Each carries a severity multiplier that feeds\ninto the penalty formula.
\nTrigger: More than 15 occurrences of MUST, ALWAYS, or NEVER in the SKILL.md.
\nProblem: Overly prescriptive instructions reduce model flexibility, increase token overhead,\nand signal that the author is trying to micromanage every output rather than providing\nprincipled guidance.
\nFix: Audit every MUST/ALWAYS/NEVER. Replace directive language with explanatory framing\nwhere possible. Reserve hard constraints for genuine safety or correctness requirements. Target\nfewer than 10 such directives per 100 lines.
\nTrigger: The frontmatter description field is fewer than 20 characters after stripping.
Problem: Without a meaningful description, the Claude Code plugin system cannot determine\nwhen to invoke the skill. The skill becomes invisible to autonomous invocation.
\nFix: Write a description of at least 60–120 characters that includes:
\nTrigger: The description does not contain \"use when\", \"use this skill when\",\n\"use proactively\", or \"trigger when\" (case-insensitive).
\nProblem: Even a long description is useless for autonomous invocation if it doesn't\ninclude a clear trigger signal. The system's routing model needs an explicit cue.
\nFix: Prepend \"Use this skill when...\" to the description, followed by specific scenarios.\nExample: \"Use this skill when measuring plugin quality, interpreting score reports, or\nexplaining badge thresholds to a team.\"
\nTrigger: SKILL.md exceeds 800 lines AND the skill has no references/ directory.
Problem: A monolithic SKILL.md forces the entire document into context on every invocation,\nwasting tokens on content only needed in edge cases.
\nFix: Create a references/ directory and move supporting material there:
references/rubrics.mdreferences/examples.mdreferences/config.mdThe SKILL.md should link to these files with [text](references/filename.md) so the model\ncan fetch them on demand.
Trigger: SKILL.md contains a markdown link [text](references/filename) where\nfilename does not exist in the references/ directory.
Problem: Dead links waste tokens on context that will never resolve and confuse the model.
\nFix: Either create the missing reference file or remove the dead link.
\nTrigger: SKILL.md references another skill or agent by relative path and that path\ncannot be resolved from the skills/ directory.
\nProblem: Broken ecosystem links undermine the plugin's coherence score and may cause\nthe model to attempt navigation to non-existent files.
\nFix: Verify the referenced skill exists. Update the path or remove the reference.
\nPluginEval uses an Elo/Bradley-Terry rating system to rank a skill against the gold corpus.
\nStarting rating: 1500 (the corpus median by convention).
\nK-factor: 32 (standard for moderate-stakes ratings).
\nExpected score formula (standard Elo):
\nE(A vs B) = 1 / (1 + 10^((B_rating − A_rating) / 400))\nRating update after each matchup:
\nnew_rating = old_rating + 32 × (actual_score − expected_score)\nwhere actual_score is 1.0 for a win, 0.5 for a draw, 0.0 for a loss.
Confidence intervals are computed via 500-sample bootstrap, reported as 95% CI.\nCorpus percentile reflects pairwise win rate against the gold corpus.\nPosition bias check: Pairs are evaluated in both orders; disagreements are flagged.
\nThe plugin-eval init command builds the corpus index from a plugins directory:
plugin-eval init ./plugins --corpus-dir ~/.plugineval/corpus\nplugin-eval score ./path/to/skill --depth quick\nReturns Layer 1 results in < 2 seconds. Useful for fast feedback during authoring.
\nplugin-eval score ./path/to/skill\nRuns static + LLM judge (standard depth). Takes 30–90 seconds.
\nplugin-eval score ./path/to/skill --output json\nEmits structured JSON including composite.score, composite.dimensions, and\nlayers[0].anti_patterns. Suitable for CI integration:
plugin-eval score ./path/to/skill --depth quick --output json --threshold 70\n# exits with code 1 if score < 70\nplugin-eval certify ./path/to/skill\nRuns static + LLM judge + Monte Carlo (50 simulations) + Elo ranking. Takes 15–20 minutes.\nAssigns a quality badge. Use before publishing a skill to the marketplace.
\nplugin-eval compare ./skill-a ./skill-b\nEvaluates both skills at quick depth and prints a dimension-by-dimension comparison table.\nUseful for deciding between two implementations or measuring improvement before/after a\nrewrite.
\nplugin-eval init ./plugins\nBuilds the local corpus index at ~/.plugineval/corpus. Required before Elo ranking works.
Reproduce the composite score offline (pre-commit hook, CI gate):
\ndef composite_score(dimension_scores: dict, anti_pattern_count: int = 0) -> float:\n \"\"\"Replicate the PluginEval composite formula.\"\"\"\n WEIGHTS = {\n \"triggering_accuracy\": 0.25,\n \"orchestration_fitness\": 0.20,\n \"output_quality\": 0.15,\n \"scope_calibration\": 0.12,\n \"progressive_disclosure\": 0.10,\n \"token_efficiency\": 0.06,\n \"robustness\": 0.05,\n \"structural_completeness\":0.03,\n \"code_template_quality\": 0.02,\n \"ecosystem_coherence\": 0.02,\n }\n raw = sum(WEIGHTS[d] * s for d, s in dimension_scores.items())\n penalty = max(0.5, 1.0 - 0.05 * anti_pattern_count)\n return round(raw * 100 * penalty, 2)\n\n# Example: a skill with a weak triggering score\nscores = {\n \"triggering_accuracy\": 0.65, # D — needs description work\n \"orchestration_fitness\": 0.85,\n \"output_quality\": 0.80,\n # … fill in remaining 7 dimensions …\n}\n# composite_score(scores, anti_pattern_count=1) → ~76.5\nTop-level shape of --output json:
{\n \"composite\": { \"score\": 76.5, \"badge\": \"Silver\", \"elo\": null },\n \"dimensions\": {\n \"triggering_accuracy\": { \"score\": 0.65, \"grade\": \"D\", \"ci_low\": 0.60, \"ci_high\": 0.70 },\n \"orchestration_fitness\": { \"score\": 0.85, \"grade\": \"B\", \"ci_low\": 0.80, \"ci_high\": 0.90 }\n },\n \"layers\": [\n { \"name\": \"static\", \"duration_ms\": 1243, \"anti_patterns\": [\"OVER_CONSTRAINED\"] },\n { \"name\": \"judge\", \"duration_ms\": 48200, \"judges\": 1, \"kappa\": null }\n ]\n}\nParse composite.score in CI to gate deployments:
score=$(plugin-eval score ./my-skill --output json | python3 -c \"import sys,json; print(json.load(sys.stdin)['composite']['score'])\")\nif (( $(echo \"$score < 70\" | bc -l) )); then\n echo \"Quality gate failed: score $score < 70\"\n exit 1\nfi\nWork through dimensions in weight order. The largest gains come from fixing the top-weighted\ndimensions first.
\nUse this table when a score report shows multiple D/F grades and you need to prioritize effort.
\n| Dimension | \nWeight | \nTypical fix effort | \nScore impact / hour | \nFix first if… | \n
|---|---|---|---|---|
triggering_accuracy | \n0.25 | \nLow — description rewrite | \nHigh | \nScore < 70 overall | \n
orchestration_fitness | \n0.20 | \nMedium — restructure sections | \nHigh | \nSkill mixes worker + supervisor logic | \n
output_quality | \n0.15 | \nMedium — add examples | \nMedium | \nJudge score < 0.70 | \n
scope_calibration | \n0.12 | \nLow — move content to references/ | \nMedium | \nFile is < 100 or > 800 lines | \n
progressive_disclosure | \n0.10 | \nLow — create references/ dir | \nMedium | \nNo references/ directory exists | \n
token_efficiency | \n0.06 | \nLow — reduce MUST/ALWAYS/NEVER | \nLow | \nAnti-pattern count ≥ 3 | \n
robustness | \n0.05 | \nLow — add Troubleshooting section | \nLow | \nNo edge-case handling documented | \n
structural_completeness | \n0.03 | \nVery low — add headings/code blocks | \nLow | \nFewer than 4 H2 headings | \n
code_template_quality | \n0.02 | \nVery low — add language tags | \nVery low | \nCode blocks missing language tags | \n
ecosystem_coherence | \n0.02 | \nVery low — add Related section | \nVery low | \nNo cross-references at all | \n
Rule of thumb: Fix triggering_accuracy before anything else — at weight 0.25 it delivers\nmore composite-score gain per hour than all low-weight dimensions combined.
references/ is bloat.references/.references/ directory (earns 0.15–0.25 bonus) and keep SKILL.md focused on\nthe execution path. An assets/ directory adds a further bonus.The anti-pattern penalty compounds. Run with --output json and inspect\nlayers[0].anti_patterns. If you have 5+ anti-patterns, the multiplier can reduce your\nscore to 75% of its raw value regardless of how good the content is. Fix the flags first.
The _description_pushiness scorer looks for specific syntactic patterns, not just length.\nVerify your description contains the phrase \"Use this skill when\" or \"Use when\" (exact\nphrasing matters — it's a regex match). Also check that you have multiple use cases separated\nby commas or \"or\" to earn the specificity bonus.
This is expected for ambiguous skills. The judge generates 10 mental test prompts\nnon-deterministically. Improve score stability by tightening the description and adding\nconcrete examples. When judges > 1, averaged scores will be more stable. Use\n--depth deep with certify which runs Monte Carlo to get statistically-bounded scores.
Check whether the file is in the 200–600 line sweet spot. Files shorter than 100 lines\nscore only 0.20 on this sub-check. Also confirm that references/ files are not empty —\nthe scorer checks for non-empty reference files, not just the directory.
Quick depth (--depth quick) only runs static analysis. If the rewrite moved content to\nreferences/ and shortened SKILL.md significantly, static scores for structural completeness\nmay drop even though overall quality improved. Run --depth standard for a fairer comparison\nthat includes the LLM judge's assessment of content quality.
../../agents/eval-judge.md) — the LLM judge that scores Layer 2 dimensions\n(triggering_accuracy, orchestration_fitness, output_quality, scope_calibration).\nInvoke directly when you need to re-run only the judge layer or inspect its reasoning.../../agents/eval-orchestrator.md) — the top-level orchestrator that\nsequences all three layers, merges results, assigns badges, and writes the final report.\nInvoke when running a full certification pass or comparing two skills head-to-head.