Closed-loop skill effectiveness monitoring. Reads session metrics,\ncomputes per-skill signals, identifies what's working and what needs\nimprovement.
\nInspired by the deploy-monitor-evaluate-improve feedback loop:\nskills get better over time instead of staying static.
\nRequires .claude/session-metrics/metrics.jsonl from /session-scan.\nIf no data: suggest running /session-scan first.
/skill-monitor # Dashboard: all skills\n/skill-monitor --skill review # Deep-dive on one skill\n/skill-monitor --improve # Generate improvement recommendations\n/skill-monitor --window 30d # Change comparison window (default: 7d)\nExtract from $ARGUMENTS:
--skill NAME: Focus on one skill (e.g., review, plan, investigate)--improve: Spawn analysis agent for improvement recommendations--window PERIOD: Comparison window (7d, 30d, all; default: 7d)Read .claude/session-metrics/metrics.jsonl. For each entry, extract\nthe skill_effectiveness field (added by compute-metrics.py v2).
Filter by window period. Count sessions with and without skill usage.
\nIf no skill_effectiveness data exists in metrics: \"Metrics were\ncomputed before skill tracking was added. Run /session-scan --rescan\nto recompute.\"
OTel invocation_trigger (CC v2.1.126+): when compute-metrics.py\ningests claude_code.skill_activated events, each invocation carries\nan invocation_trigger of \"user-slash\", \"claude-proactive\", or\n\"nested-skill\". If absent (older sessions), default to\n\"unknown\" — do NOT assume \"user-slash\".
For each skill found across all sessions, aggregate:
\n| Metric | Computation |\n|-------------------------|------------------------------------------------|\n| Total invocations | Sum of invocation_count across sessions |\n| Sessions used in | Count of sessions containing this skill |\n| Action rate | Weighted avg of per-session action_rate |\n| Avg post-errors | Weighted avg of avg_post_errors |\n| Avg post-corrections | Weighted avg of avg_post_corrections |\n| Outcome distribution | Count of effective/friction/no_action/mixed |\n| Effectiveness score | action_rate - (0.3 * avg_post_corrections) |\n| Adjusted score | For analysis/check skills, use lower thresholds |\n| Trigger distribution | Counts of user-slash / claude-proactive / nested-skill / unknown |\n| Proactive trigger rate | claude-proactive / (user-slash + claude-proactive + nested-skill) |\n| Auto-load gap | Skills with 0 claude-proactive invocations across window |\nAuto-load gap detection (CC v2.1.126+): Skills with auto-loaded\nbehavior in their description (i.e., not disable-model-invocation: true)\nare EXPECTED to fire as claude-proactive. A skill that is ONLY ever\ninvoked via user-slash is failing its description's routing intent.\nFlag any auto-loadable skill where proactive_trigger_rate == 0 over\nthe window. This is the structural answer to the \"zero skill\nauto-loading\" gap from the 137-session analysis (see MEMORY.md).\nConfidence floor: only flag if total invocations >= 5 in window.
Skill type weighting: Analysis and check skills (verify, triage,\nperf, boundaries, pr-review, audit) have low action rates BY DESIGN —\ntheir success is \"found issues\" or \"confirmed things pass\". Apply\nadjusted thresholds:
\n| Skill Type | \nFlag Threshold | \nExpected Action Rate | \n
|---|---|---|
| Execution (work, quick, full) | \n< 0.5 | \n> 0.7 | \n
| Analysis (perf, boundaries, audit, pr-review) | \n< 0.3 | \n0.3-0.5 | \n
| Check (verify, triage) | \n< 0.1 | \n0.0-0.3 | \n
| Knowledge (compound, learn, brief) | \n< 0.5 | \n> 0.5 | \n
Also compute baseline friction (avg friction of sessions WITHOUT\nany skill usage) vs skill friction (avg friction of sessions\nWITH skill usage). Delta = skill_friction - baseline_friction.\nNegative delta = skills reduce friction (good).
\nDashboard mode (no --skill):
## Skill Effectiveness Dashboard (last {window})\n\nBaseline friction (no skills): 0.32 | With skills: 0.18 | Delta: -0.14\n\n| Skill | Uses | Sessions | Slash/Proactive/Nested | Action% | Errors | Corr | Outcome | Score |\n|-----------------|------|----------|------------------------|---------|--------|------|-----------|-------|\n| /phx:review | 12 | 8 | 8 / 3 / 1 | 92% | 0.5 | 0.1 | effective | 0.89 |\n| /phx:plan | 9 | 7 | 9 / 0 / 0 | 100% | 0.2 | 0.0 | effective | 1.00 |\n| /phx:investigate| 5 | 5 | 5 / 0 / 0 | 80% | 1.2 | 0.4 | mixed | 0.68 |\n\nSkills needing attention:\n- /phx:investigate (high post-errors)\n- /phx:plan (auto-load gap — 0/9 proactive; description not routing)\nFlag skills using type-adjusted thresholds (see weighting table above).\nAlso flag if avg_post_corrections > 1 or outcome is predominantly \"friction\".\nAlso flag auto-load gap: auto-loadable skills (without\ndisable-model-invocation: true) with proactive_trigger_rate == 0 and\ntotal invocations >= 5. This is a description/routing problem — the skill\nexists but Claude isn't loading it on its own.
When displaying flagged skills, note if the flag is \"expected\" for the\nskill type (e.g., verify at 0.24 is normal for a check skill).
\nSkill deep-dive (--skill NAME):
Show per-session breakdown for that skill, including session IDs,\ndates, individual outcome signals, AND invocation_trigger per\ninvocation. If a skill is dominated by user-slash triggers, surface\nwhich 1-3 description keywords might unlock proactive routing —\ncross-reference against the skill's current description in\nplugins/elixir-phoenix/skills/{name}/SKILL.md. If session reports\nexist in .claude/session-analysis/, reference them.
Spawn skill-effectiveness-analyzer agent:
Agent(subagent_type=\"skill-effectiveness-analyzer\", model=\"sonnet\", prompt=\"\"\"\nAnalyze skill effectiveness data and recommend improvements.\n\nMetrics data: {aggregated_metrics_json}\n\nSessions with friction outcomes: {session_ids}\n\nFor each underperforming skill:\n1. Identify failure patterns from outcome signals\n2. Propose specific skill/agent changes\n3. Suggest new Iron Laws if patterns are systematic\n\nWrite recommendations to: .claude/skill-metrics/recommendations-{date}.md\n\"\"\")\nWrite aggregated metrics to .claude/skill-metrics/dashboard-{date}.json:
{\n \"computed_at\": \"2026-03-03T14:00:00Z\",\n \"window\": \"7d\",\n \"baseline_friction\": 0.32,\n \"skill_friction\": 0.18,\n \"friction_delta\": -0.14,\n \"skills\": {\n \"/phx:plan\": {\n \"invocations\": 9,\n \"trigger_distribution\": {\n \"user-slash\": 9,\n \"claude-proactive\": 0,\n \"nested-skill\": 0,\n \"unknown\": 0\n },\n \"proactive_trigger_rate\": 0.0,\n \"auto_load_gap\": true\n }\n },\n \"flagged_skills\": [\"investigate\", \"plan:auto-load-gap\"]\n}\nAppend-only: never modify previous dashboard files.
\nuser-slash / claude-proactive / nested-skill when the OTel\ninvocation_trigger attribute is present (CC v2.1.126+). Older\nsessions use \"unknown\"; never silently bucket them as user-slash —\nit would hide the auto-load gap./session-scan → metrics.jsonl (with skill_effectiveness)\n ↓\n/skill-monitor → dashboard + flagged skills\n ↓\n/skill-monitor --improve → recommendations\n ↓\nDeveloper updates skills/agents → deploy → repeat\nreferences/effectiveness-metrics.md — Full metrics schema and evaluation criteriareferences/improvement-template.md — Template for improvement recommendations