Claude Skill

prompt-regression-testing

Builds and runs regression tests for prompts, skills and agents so changes can be checked instead of guessed. Use whenever the user changes a prompt or SKILL.md and wants to know if it got better or worse, asks how to test or evaluate an LLM feature, wants a frozen test set, pass

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Download mehranqadirian-skills-agent-engineering_prompt-regression-testing-c88187d.zip · 6 KB

Install

skills CLI npx skills add https://github.com/MehranQadirian/skills/tree/main/agent-engineering/prompt-regression-testing
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mehranqadirian-skills@llmmart
Git git clone https://github.com/MehranQadirian/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mehranqadirian/skills collection as a plugin from our marketplace. Git is the plain clone.

README

prompt-regression-testing

A Claude skill for testing prompts, skills and agents with a frozen case set, deterministic checks and baseline comparison.

What it does

  • Runs your cases through any command or recorded answers
  • Checks text, patterns, length and JSON structure
  • Repeats runs and applies a pass rate threshold for non-deterministic outputs
  • Compares with a saved baseline and flags regressions, fixes and new cases
  • Provides a case design guide and an example suite

Try it

I edited my prompt. Help me check it did not get worse.

Build a test suite for this skill, including a case that should not trigger it.

Compare version A and version B of my system prompt.

Install

Claude.ai: zip this folder, then go to Settings > Capabilities > Skills > Upload.

Claude Code:

cp -r agent-engineering/prompt-regression-testing ~/.claude/skills/

Structure

prompt-regression-testing/
├── SKILL.md                       skill instructions for Claude
├── scripts/run_cases.py           test runner with baseline diff
├── references/case-design.md      designing cases and checks
└── templates/cases.json           example suite

Skill manifest

prompt-regression-testing

A change to a prompt, skill or agent can fix one case and silently break three. Freeze a small set of realistic cases, check them with deterministic rules, and compare every change to a saved baseline.

prompt-regression-testing/
├── SKILL.md
├── scripts/
│   └── run_cases.py          run cases via a command or recorded answers; checks, repeats, baseline diff
├── references/
│   └── case-design.md        coverage map, check types, non-determinism, subjective quality, CI
└── templates/
    └── cases.json            example suite with typical, format, safety and negative cases

When to use

  • Editing a prompt, SKILL.md or agent and needing proof it did not get worse
  • Choosing between two prompt versions
  • Setting up checks before shipping an LLM feature
  • A bug was fixed and must never return

Workflow

  1. Collect cases from real requests. Cover typical, edge, format, safety, and at least one negative case that should not trigger the behavior (references/case-design.md). Start with about 20.
  2. Write checks per case in cases.json (copy templates/cases.json). Prefer several loose checks over one strict check.
  3. Write a runner: reads the input from stdin, prints the answer to stdout. Keep keys in environment variables.
  4. Freeze a baseline on the version you trust: python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --save baseline.json
  5. Change the prompt, then compare: python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --baseline baseline.json Any [REGRESSION] line means a case that passed before now fails.
  6. Handle variation with --repeat 10 --min-pass-rate 0.9 for non-deterministic outputs.
  7. Decide: keep the change only if there are no regressions, or the trade is explicit and accepted. Save the new baseline and commit cases with the change.
  8. Add a case for every bug you fix.

Rules

  • Never edit a frozen case to make it pass; add a new one
  • Write cases before looking at the new output, or the test will bend to the answer
  • One behavior per case, so failures point to a cause
  • Do not lower the pass rate to hide flakiness; rewrite the prompt
  • Use recorded answers (--outputs) in CI and run live checks on a schedule
  • Deterministic checks cannot judge style; for that use a rubric with spot checks by hand

Commands

python3 scripts/run_cases.py cases.json --cmd "python3 runner.py"
python3 scripts/run_cases.py cases.json --outputs answers.json
python3 scripts/run_cases.py cases.json --cmd "..." --repeat 10 --min-pass-rate 0.9 --save results.json

Exit code is 0 when every case meets the pass rate and nothing regressed, 1 otherwise.

What to report back

Field Content
Suite number of cases and kinds covered
Result passed over total, per failing case with the reason
Comparison regressions, fixes and new cases versus the baseline
Variation repeats and pass rates for flaky cases
Decision keep, revert or fix, with the evidence

Related skills

  • skill-authoring: create or fix the skill that these tests protect
  • multi-agent-orchestration: evaluate agent systems against a single-agent baseline
  • systematic-debugging: trace a failing case to its cause
Files (skills)
  • references
    • case-design.md 3.3 KB
      Loading…
  • scripts
    • run_cases.py 5.3 KB
      Loading…
  • templates
    • cases.json 960 B
      Loading…
  • README.md 1.2 KB
    # prompt-regression-testing
    
    A [Claude skill](../../README.md) for testing prompts, skills and agents with a frozen case set, deterministic checks and baseline comparison.
    
    ## What it does
    
    - Runs your cases through any command or recorded answers
    - Checks text, patterns, length and JSON structure
    - Repeats runs and applies a pass rate threshold for non-deterministic outputs
    - Compares with a saved baseline and flags regressions, fixes and new cases
    - Provides a case design guide and an example suite
    
    ## Try it
    
    > I edited my prompt. Help me check it did not get worse.
    
    > Build a test suite for this skill, including a case that should not trigger it.
    
    > Compare version A and version B of my system prompt.
    
    ## Install
    
    **Claude.ai:** zip this folder, then go to Settings > Capabilities > Skills > Upload.
    
    **Claude Code:**
    
    ```bash
    cp -r agent-engineering/prompt-regression-testing ~/.claude/skills/
    ```
    
    ## Structure
    
    ```
    prompt-regression-testing/
    ├── SKILL.md                       skill instructions for Claude
    ├── scripts/run_cases.py           test runner with baseline diff
    ├── references/case-design.md      designing cases and checks
    └── templates/cases.json           example suite
    ```
    
  • SKILL.md 3.8 KB
    ---
    name: prompt-regression-testing
    description: Builds and runs regression tests for prompts, skills and agents so changes can be checked instead of guessed. Use whenever the user changes a prompt or SKILL.md and wants to know if it got better or worse, asks how to test or evaluate an LLM feature, wants a frozen test set, pass rates over repeated runs, a baseline comparison or CI checks for prompts, even if they only say "did my edit break anything". Model agnostic, deterministic checks, flags regressions.
    ---
    
    # prompt-regression-testing
    
    A change to a prompt, skill or agent can fix one case and silently break three. Freeze a small set of realistic cases, check them with deterministic rules, and compare every change to a saved baseline.
    
    ```
    prompt-regression-testing/
    ├── SKILL.md
    ├── scripts/
    │   └── run_cases.py          run cases via a command or recorded answers; checks, repeats, baseline diff
    ├── references/
    │   └── case-design.md        coverage map, check types, non-determinism, subjective quality, CI
    └── templates/
        └── cases.json            example suite with typical, format, safety and negative cases
    ```
    
    ## When to use
    
    - Editing a prompt, SKILL.md or agent and needing proof it did not get worse
    - Choosing between two prompt versions
    - Setting up checks before shipping an LLM feature
    - A bug was fixed and must never return
    
    ## Workflow
    
    1. **Collect cases** from real requests. Cover typical, edge, format, safety, and at least one negative case that should not trigger the behavior (`references/case-design.md`). Start with about 20.
    2. **Write checks** per case in `cases.json` (copy `templates/cases.json`). Prefer several loose checks over one strict check.
    3. **Write a runner**: reads the input from stdin, prints the answer to stdout. Keep keys in environment variables.
    4. **Freeze a baseline** on the version you trust:
       `python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --save baseline.json`
    5. **Change the prompt, then compare**:
       `python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --baseline baseline.json`
       Any `[REGRESSION]` line means a case that passed before now fails.
    6. **Handle variation** with `--repeat 10 --min-pass-rate 0.9` for non-deterministic outputs.
    7. **Decide**: keep the change only if there are no regressions, or the trade is explicit and accepted. Save the new baseline and commit cases with the change.
    8. **Add a case** for every bug you fix.
    
    ## Rules
    
    - Never edit a frozen case to make it pass; add a new one
    - Write cases before looking at the new output, or the test will bend to the answer
    - One behavior per case, so failures point to a cause
    - Do not lower the pass rate to hide flakiness; rewrite the prompt
    - Use recorded answers (`--outputs`) in CI and run live checks on a schedule
    - Deterministic checks cannot judge style; for that use a rubric with spot checks by hand
    
    ## Commands
    
    ```bash
    python3 scripts/run_cases.py cases.json --cmd "python3 runner.py"
    python3 scripts/run_cases.py cases.json --outputs answers.json
    python3 scripts/run_cases.py cases.json --cmd "..." --repeat 10 --min-pass-rate 0.9 --save results.json
    ```
    
    Exit code is 0 when every case meets the pass rate and nothing regressed, 1 otherwise.
    
    ## What to report back
    
    | Field | Content |
    |-------|---------|
    | Suite | number of cases and kinds covered |
    | Result | passed over total, per failing case with the reason |
    | Comparison | regressions, fixes and new cases versus the baseline |
    | Variation | repeats and pass rates for flaky cases |
    | Decision | keep, revert or fix, with the evidence |
    
    ## Related skills
    
    - `skill-authoring`: create or fix the skill that these tests protect
    - `multi-agent-orchestration`: evaluate agent systems against a single-agent baseline
    - `systematic-debugging`: trace a failing case to its cause
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.