bounded-evaluation
Design small evaluation gates for agent outputs, skill changes, prompts, and tool contracts. Use when a change needs rubric checks, pairwise comparison, position-bias mitigation, regression fixtures, or LLM-as-judge discipline without building a heavy eval platform.
Install
npx skills add https://github.com/markoblogo/abvx-agent-skills/tree/main/skills/bounded-evaluation
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install markoblogo-abvx-agent-skills@llmmart
git clone https://github.com/markoblogo/abvx-agent-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole markoblogo/abvx-agent-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Bounded Evaluation
Use this skill when a change needs a small, explicit evaluation gate.
Use When
- reviewing skill changes, AGENTS.md rules, prompt playbooks, or SET bundle changes;
- comparing two agent outputs or proposals;
- using an LLM judge where bias or weak rubrics could mislead;
- creating activation tests, regression cases, or ship gates for agent behavior;
- deciding whether a repeated workflow improved after a bounded edit.
Evaluation Contract
A bounded eval must state:
- target artifact or behavior;
- candidate outputs or versions;
- rubric dimensions and weights, if any;
- pass/fail threshold;
- fixtures or scenarios;
- judge type: deterministic check, human review, LLM judge, or hybrid;
- known bias risks and mitigation.
Pairwise And Judge Rules
- Compare A/B outputs against the same rubric.
- Swap candidate order when using an LLM judge to detect position bias.
- Require reasons before scores, but treat reasons as evidence to inspect, not truth.
- Mark the eval invalid when swapped order changes the winner without a defensible reason.
- Prefer deterministic checks for syntax, schema, file existence, and command success.
- Keep human approval for high-stakes or ambiguous decisions.
Workflow
- Define the smallest behavior being evaluated.
- Pick 1-5 representative fixtures or scenarios.
- Choose deterministic checks before LLM judging.
- If using pairwise judging, run both candidate orders.
- Record results, invalidations, and decision.
- Feed accepted/rejected outcomes back into the relevant ledger.
Output Shape
Use:
- eval target;
- fixtures;
- rubric;
- judge type;
- results;
- bias checks;
- decision:
accept,reject,revise, orinvalid; - follow-up artifact or ledger update.
Guardrails
- Do not publish benchmark claims without committed fixtures, arms, model, host, run count, and captured artifacts.
- Do not let an LLM judge approve its own rubric changes.
- Do not use pairwise preference as proof of correctness.
- Do not hide invalid or inconsistent judge results.
Provenance
Adapted from advanced-evaluation patterns in muratcankoylan/Agent-Skills-for-Context-Engineering, aligned with ABVX SkillOpt validation gates.
Files (abvx-agent-skills)
-
agents
-
openai.yaml 214 B
Loading…
-
-
SKILL.md 2.6 KB
--- name: bounded-evaluation description: Design small evaluation gates for agent outputs, skill changes, prompts, and tool contracts. Use when a change needs rubric checks, pairwise comparison, position-bias mitigation, regression fixtures, or LLM-as-judge discipline without building a heavy eval platform. license: MIT metadata: abvx_status: experimental abvx_origin: adapted --- # Bounded Evaluation Use this skill when a change needs a small, explicit evaluation gate. ## Use When - reviewing skill changes, AGENTS.md rules, prompt playbooks, or SET bundle changes; - comparing two agent outputs or proposals; - using an LLM judge where bias or weak rubrics could mislead; - creating activation tests, regression cases, or ship gates for agent behavior; - deciding whether a repeated workflow improved after a bounded edit. ## Evaluation Contract A bounded eval must state: - target artifact or behavior; - candidate outputs or versions; - rubric dimensions and weights, if any; - pass/fail threshold; - fixtures or scenarios; - judge type: deterministic check, human review, LLM judge, or hybrid; - known bias risks and mitigation. ## Pairwise And Judge Rules - Compare A/B outputs against the same rubric. - Swap candidate order when using an LLM judge to detect position bias. - Require reasons before scores, but treat reasons as evidence to inspect, not truth. - Mark the eval invalid when swapped order changes the winner without a defensible reason. - Prefer deterministic checks for syntax, schema, file existence, and command success. - Keep human approval for high-stakes or ambiguous decisions. ## Workflow 1. Define the smallest behavior being evaluated. 2. Pick 1-5 representative fixtures or scenarios. 3. Choose deterministic checks before LLM judging. 4. If using pairwise judging, run both candidate orders. 5. Record results, invalidations, and decision. 6. Feed accepted/rejected outcomes back into the relevant ledger. ## Output Shape Use: - eval target; - fixtures; - rubric; - judge type; - results; - bias checks; - decision: `accept`, `reject`, `revise`, or `invalid`; - follow-up artifact or ledger update. ## Guardrails - Do not publish benchmark claims without committed fixtures, arms, model, host, run count, and captured artifacts. - Do not let an LLM judge approve its own rubric changes. - Do not use pairwise preference as proof of correctness. - Do not hide invalid or inconsistent judge results. ## Provenance Adapted from advanced-evaluation patterns in `muratcankoylan/Agent-Skills-for-Context-Engineering`, aligned with ABVX SkillOpt validation gates. -
SKILL_CARD.md 1.3 KB
Loading…
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.