prompt-regression-testing
Builds and runs regression tests for prompts, skills and agents so changes can be checked instead of guessed. Use whenever the user changes a prompt or SKILL.md and wants to know if it got better or worse, asks how to test or evaluate an LLM feature, wants a frozen test set, pass
Install
npx skills add https://github.com/MehranQadirian/skills/tree/main/agent-engineering/prompt-regression-testing
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mehranqadirian-skills@llmmart
git clone https://github.com/MehranQadirian/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mehranqadirian/skills collection as a plugin from our marketplace. Git is the plain clone.
README
prompt-regression-testing
A Claude skill for testing prompts, skills and agents with a frozen case set, deterministic checks and baseline comparison.
What it does
- Runs your cases through any command or recorded answers
- Checks text, patterns, length and JSON structure
- Repeats runs and applies a pass rate threshold for non-deterministic outputs
- Compares with a saved baseline and flags regressions, fixes and new cases
- Provides a case design guide and an example suite
Try it
I edited my prompt. Help me check it did not get worse.
Build a test suite for this skill, including a case that should not trigger it.
Compare version A and version B of my system prompt.
Install
Claude.ai: zip this folder, then go to Settings > Capabilities > Skills > Upload.
Claude Code:
cp -r agent-engineering/prompt-regression-testing ~/.claude/skills/
Structure
prompt-regression-testing/
├── SKILL.md skill instructions for Claude
├── scripts/run_cases.py test runner with baseline diff
├── references/case-design.md designing cases and checks
└── templates/cases.json example suite
Skill manifest
prompt-regression-testing
A change to a prompt, skill or agent can fix one case and silently break three. Freeze a small set of realistic cases, check them with deterministic rules, and compare every change to a saved baseline.
prompt-regression-testing/
├── SKILL.md
├── scripts/
│ └── run_cases.py run cases via a command or recorded answers; checks, repeats, baseline diff
├── references/
│ └── case-design.md coverage map, check types, non-determinism, subjective quality, CI
└── templates/
└── cases.json example suite with typical, format, safety and negative cases
When to use
- Editing a prompt, SKILL.md or agent and needing proof it did not get worse
- Choosing between two prompt versions
- Setting up checks before shipping an LLM feature
- A bug was fixed and must never return
Workflow
- Collect cases from real requests. Cover typical, edge, format, safety, and at least one negative case that should not trigger the behavior (
references/case-design.md). Start with about 20. - Write checks per case in
cases.json(copytemplates/cases.json). Prefer several loose checks over one strict check. - Write a runner: reads the input from stdin, prints the answer to stdout. Keep keys in environment variables.
- Freeze a baseline on the version you trust:
python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --save baseline.json - Change the prompt, then compare:
python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --baseline baseline.jsonAny[REGRESSION]line means a case that passed before now fails. - Handle variation with
--repeat 10 --min-pass-rate 0.9for non-deterministic outputs. - Decide: keep the change only if there are no regressions, or the trade is explicit and accepted. Save the new baseline and commit cases with the change.
- Add a case for every bug you fix.
Rules
- Never edit a frozen case to make it pass; add a new one
- Write cases before looking at the new output, or the test will bend to the answer
- One behavior per case, so failures point to a cause
- Do not lower the pass rate to hide flakiness; rewrite the prompt
- Use recorded answers (
--outputs) in CI and run live checks on a schedule - Deterministic checks cannot judge style; for that use a rubric with spot checks by hand
Commands
python3 scripts/run_cases.py cases.json --cmd "python3 runner.py"
python3 scripts/run_cases.py cases.json --outputs answers.json
python3 scripts/run_cases.py cases.json --cmd "..." --repeat 10 --min-pass-rate 0.9 --save results.json
Exit code is 0 when every case meets the pass rate and nothing regressed, 1 otherwise.
What to report back
| Field | Content |
|---|---|
| Suite | number of cases and kinds covered |
| Result | passed over total, per failing case with the reason |
| Comparison | regressions, fixes and new cases versus the baseline |
| Variation | repeats and pass rates for flaky cases |
| Decision | keep, revert or fix, with the evidence |
Related skills
skill-authoring: create or fix the skill that these tests protectmulti-agent-orchestration: evaluate agent systems against a single-agent baselinesystematic-debugging: trace a failing case to its cause
Files (skills)
-
references
-
case-design.md 3.3 KB
Loading…
-
-
scripts
-
run_cases.py 5.3 KB
Loading…
-
-
templates
-
cases.json 960 B
Loading…
-
-
README.md 1.2 KB
# prompt-regression-testing A [Claude skill](../../README.md) for testing prompts, skills and agents with a frozen case set, deterministic checks and baseline comparison. ## What it does - Runs your cases through any command or recorded answers - Checks text, patterns, length and JSON structure - Repeats runs and applies a pass rate threshold for non-deterministic outputs - Compares with a saved baseline and flags regressions, fixes and new cases - Provides a case design guide and an example suite ## Try it > I edited my prompt. Help me check it did not get worse. > Build a test suite for this skill, including a case that should not trigger it. > Compare version A and version B of my system prompt. ## Install **Claude.ai:** zip this folder, then go to Settings > Capabilities > Skills > Upload. **Claude Code:** ```bash cp -r agent-engineering/prompt-regression-testing ~/.claude/skills/ ``` ## Structure ``` prompt-regression-testing/ ├── SKILL.md skill instructions for Claude ├── scripts/run_cases.py test runner with baseline diff ├── references/case-design.md designing cases and checks └── templates/cases.json example suite ``` -
SKILL.md 3.8 KB
--- name: prompt-regression-testing description: Builds and runs regression tests for prompts, skills and agents so changes can be checked instead of guessed. Use whenever the user changes a prompt or SKILL.md and wants to know if it got better or worse, asks how to test or evaluate an LLM feature, wants a frozen test set, pass rates over repeated runs, a baseline comparison or CI checks for prompts, even if they only say "did my edit break anything". Model agnostic, deterministic checks, flags regressions. --- # prompt-regression-testing A change to a prompt, skill or agent can fix one case and silently break three. Freeze a small set of realistic cases, check them with deterministic rules, and compare every change to a saved baseline. ``` prompt-regression-testing/ ├── SKILL.md ├── scripts/ │ └── run_cases.py run cases via a command or recorded answers; checks, repeats, baseline diff ├── references/ │ └── case-design.md coverage map, check types, non-determinism, subjective quality, CI └── templates/ └── cases.json example suite with typical, format, safety and negative cases ``` ## When to use - Editing a prompt, SKILL.md or agent and needing proof it did not get worse - Choosing between two prompt versions - Setting up checks before shipping an LLM feature - A bug was fixed and must never return ## Workflow 1. **Collect cases** from real requests. Cover typical, edge, format, safety, and at least one negative case that should not trigger the behavior (`references/case-design.md`). Start with about 20. 2. **Write checks** per case in `cases.json` (copy `templates/cases.json`). Prefer several loose checks over one strict check. 3. **Write a runner**: reads the input from stdin, prints the answer to stdout. Keep keys in environment variables. 4. **Freeze a baseline** on the version you trust: `python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --save baseline.json` 5. **Change the prompt, then compare**: `python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" --baseline baseline.json` Any `[REGRESSION]` line means a case that passed before now fails. 6. **Handle variation** with `--repeat 10 --min-pass-rate 0.9` for non-deterministic outputs. 7. **Decide**: keep the change only if there are no regressions, or the trade is explicit and accepted. Save the new baseline and commit cases with the change. 8. **Add a case** for every bug you fix. ## Rules - Never edit a frozen case to make it pass; add a new one - Write cases before looking at the new output, or the test will bend to the answer - One behavior per case, so failures point to a cause - Do not lower the pass rate to hide flakiness; rewrite the prompt - Use recorded answers (`--outputs`) in CI and run live checks on a schedule - Deterministic checks cannot judge style; for that use a rubric with spot checks by hand ## Commands ```bash python3 scripts/run_cases.py cases.json --cmd "python3 runner.py" python3 scripts/run_cases.py cases.json --outputs answers.json python3 scripts/run_cases.py cases.json --cmd "..." --repeat 10 --min-pass-rate 0.9 --save results.json ``` Exit code is 0 when every case meets the pass rate and nothing regressed, 1 otherwise. ## What to report back | Field | Content | |-------|---------| | Suite | number of cases and kinds covered | | Result | passed over total, per failing case with the reason | | Comparison | regressions, fixes and new cases versus the baseline | | Variation | repeats and pass rates for flaky cases | | Decision | keep, revert or fix, with the evidence | ## Related skills - `skill-authoring`: create or fix the skill that these tests protect - `multi-agent-orchestration`: evaluate agent systems against a single-agent baseline - `systematic-debugging`: trace a failing case to its cause
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.