prompt-engineer-toolkit
Use when a marketing team's AI prompts need testing, versioning or governance, before shipping a prompt. A/B tests prompts against cases with prompt_tester.py and keeps diffed versions with prompt_versioner.py.
Install
npx skills add https://github.com/alirezarezvani/claude-skills/tree/main/marketing-skill/skills/prompt-engineer-toolkit
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install alirezarezvani-claude-skills@llmmart
git clone https://github.com/alirezarezvani/claude-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole alirezarezvani/claude-skills collection as a plugin from our marketplace. Git is the plain clone.
README
Prompt Engineer Toolkit
Production toolkit for evaluating and versioning prompts with measurable quality signals. Includes A/B testing automation and prompt history management with diffs.
Quick Start
# Run A/B prompt evaluation
python3 scripts/prompt_tester.py \
--prompt-a-file prompts/a.txt \
--prompt-b-file prompts/b.txt \
--cases-file testcases.json \
--format text
# Store a prompt version
python3 scripts/prompt_versioner.py add \
--name support_classifier \
--prompt-file prompts/a.txt \
--author team
Included Tools
scripts/prompt_tester.py: A/B testing with per-case scoring and aggregate winnerscripts/prompt_versioner.py: prompt history (add,list,diff,changelog) in local JSONL store
References
references/prompt-templates.mdreferences/technique-guide.mdreferences/evaluation-rubric.md
Installation
Claude Code
cp -R marketing-skill/prompt-engineer-toolkit ~/.claude/skills/prompt-engineer-toolkit
OpenAI Codex
cp -R marketing-skill/prompt-engineer-toolkit ~/.codex/skills/prompt-engineer-toolkit
OpenClaw
cp -R marketing-skill/prompt-engineer-toolkit ~/.openclaw/skills/prompt-engineer-toolkit
Skill manifest
Prompt Engineer Toolkit
Overview
Use this skill to move prompts from ad-hoc drafts to production assets with repeatable testing, versioning, and regression safety. It emphasizes measurable quality over intuition. Apply it when launching a new LLM feature that needs reliable outputs, when prompt quality degrades after model or instruction changes, when multiple team members edit prompts and need history/diffs, when you need evidence-based prompt choice for production rollout, or when you want consistent prompt governance across environments.
Core Capabilities
- A/B prompt evaluation against structured test cases
- Quantitative scoring for adherence, relevance, and safety checks
- Prompt version tracking with immutable history and changelog
- Prompt diffs to review behavior-impacting edits
- Reusable prompt templates and selection guidance
- Regression-friendly workflows for model/prompt updates
Key Workflows
1. Run Prompt A/B Test
Prepare JSON test cases and run:
python3 scripts/prompt_tester.py \
--prompt-a-file prompts/a.txt \
--prompt-b-file prompts/b.txt \
--cases-file testcases.json \
--runner-cmd 'my-llm-cli --prompt {prompt} --input {input}' \
--format text
Input can also come from stdin/--input JSON payload.
2. Choose Winner With Evidence
The tester scores outputs per case and aggregates:
- expected content coverage
- forbidden content violations
- regex/format compliance
- output length sanity
Use the higher-scoring prompt as candidate baseline, then run regression suite.
3. Version Prompts
# Add version
python3 scripts/prompt_versioner.py add \
--name support_classifier \
--prompt-file prompts/support_v3.txt \
--author alice
# Diff versions
python3 scripts/prompt_versioner.py diff --name support_classifier --from-version 2 --to-version 3
# Changelog
python3 scripts/prompt_versioner.py changelog --name support_classifier
4. Regression Loop
- Store baseline version.
- Propose prompt edits.
- Re-run A/B test.
- Promote only if score and safety constraints improve.
Script Interfaces
python3 scripts/prompt_tester.py --help- Reads prompts/cases from stdin or
--input - Optional external runner command
- Emits text or JSON metrics
- Reads prompts/cases from stdin or
python3 scripts/prompt_versioner.py --help- Manages prompt history (
add,list,diff,changelog) - Stores metadata and content snapshots locally
- Manages prompt history (
Pitfalls, Best Practices & Review Checklist
Avoid these mistakes:
- Picking prompts from single-case outputs — use a realistic, edge-case-rich test suite.
- Changing prompt and model simultaneously — always isolate variables.
- Missing
must_not_contain(forbidden-content) checks in evaluation criteria. - Editing prompts without version metadata, author, or change rationale.
- Skipping semantic diffs before deploying a new prompt version.
- Optimizing one benchmark while harming edge cases — track the full suite.
- Model swap without rerunning the baseline A/B suite.
Before promoting any prompt, confirm:
- Task intent is explicit and unambiguous.
- Output schema/format is explicit.
- Safety and exclusion constraints are explicit.
- No contradictory instructions.
- No unnecessary verbosity tokens.
- A/B score improves and violation count stays at zero.
References
- references/prompt-templates.md — 6 production marketing templates (ad copy, email sequence, social repurposing, landing sections, SEO meta, brand-voice rewrite) plus generic building blocks; each written to be graded by
prompt_tester.py - references/technique-guide.md — technique-selection table for marketing tasks + the LLM-governance stack for marketing teams (claim discipline, disclosure rules, data boundaries, human-review gates)
- references/evaluation-rubric.md — mechanical scoring weights, acceptance gates, marketing quality dimensions, test-suite design, and eval anti-patterns
- README.md
Evaluation Design
Each test case should define:
input: realistic production-like inputexpected_contains: required markers/contentforbidden_contains: disallowed phrases or unsafe contentexpected_regex: required structural patterns
This enables deterministic grading across prompt variants.
Versioning Policy
- Use semantic prompt identifiers per feature (
support_classifier,ad_copy_shortform). - Record author + change note for every revision.
- Never overwrite historical versions.
- Diff before promoting a new prompt to production.
Rollout Strategy
- Create baseline prompt version.
- Propose candidate prompt.
- Run A/B suite against same cases.
- Promote only if winner improves average and keeps violation count at zero.
- Track post-release feedback and feed new failure cases back into test suite.
Files (claude-skills)
-
references
-
evaluation-rubric.md 4.6 KB
Loading… -
prompt-templates.md 6 KB
Loading… -
technique-guide.md 5.8 KB
Loading…
-
-
scripts
-
prompt_tester.py 7.6 KB
Loading… -
prompt_versioner.py 7.9 KB
Loading…
-
-
README.md 1.2 KB
# Prompt Engineer Toolkit Production toolkit for evaluating and versioning prompts with measurable quality signals. Includes A/B testing automation and prompt history management with diffs. ## Quick Start ```bash # Run A/B prompt evaluation python3 scripts/prompt_tester.py \ --prompt-a-file prompts/a.txt \ --prompt-b-file prompts/b.txt \ --cases-file testcases.json \ --format text # Store a prompt version python3 scripts/prompt_versioner.py add \ --name support_classifier \ --prompt-file prompts/a.txt \ --author team ``` ## Included Tools - `scripts/prompt_tester.py`: A/B testing with per-case scoring and aggregate winner - `scripts/prompt_versioner.py`: prompt history (`add`, `list`, `diff`, `changelog`) in local JSONL store ## References - `references/prompt-templates.md` - `references/technique-guide.md` - `references/evaluation-rubric.md` ## Installation ### Claude Code ```bash cp -R marketing-skill/prompt-engineer-toolkit ~/.claude/skills/prompt-engineer-toolkit ``` ### OpenAI Codex ```bash cp -R marketing-skill/prompt-engineer-toolkit ~/.codex/skills/prompt-engineer-toolkit ``` ### OpenClaw ```bash cp -R marketing-skill/prompt-engineer-toolkit ~/.openclaw/skills/prompt-engineer-toolkit ``` -
SKILL.md 5.2 KB
--- name: "prompt-engineer-toolkit" description: "Use when a marketing team's AI prompts need testing, versioning or governance, before shipping a prompt. A/B tests prompts against cases with prompt_tester.py and keeps diffed versions with prompt_versioner.py." license: MIT metadata: version: 1.0.0 author: Alireza Rezvani category: marketing updated: 2026-03-06 --- # Prompt Engineer Toolkit ## Overview Use this skill to move prompts from ad-hoc drafts to production assets with repeatable testing, versioning, and regression safety. It emphasizes measurable quality over intuition. Apply it when launching a new LLM feature that needs reliable outputs, when prompt quality degrades after model or instruction changes, when multiple team members edit prompts and need history/diffs, when you need evidence-based prompt choice for production rollout, or when you want consistent prompt governance across environments. ## Core Capabilities - A/B prompt evaluation against structured test cases - Quantitative scoring for adherence, relevance, and safety checks - Prompt version tracking with immutable history and changelog - Prompt diffs to review behavior-impacting edits - Reusable prompt templates and selection guidance - Regression-friendly workflows for model/prompt updates ## Key Workflows ### 1. Run Prompt A/B Test Prepare JSON test cases and run: ```bash python3 scripts/prompt_tester.py \ --prompt-a-file prompts/a.txt \ --prompt-b-file prompts/b.txt \ --cases-file testcases.json \ --runner-cmd 'my-llm-cli --prompt {prompt} --input {input}' \ --format text ``` Input can also come from stdin/`--input` JSON payload. ### 2. Choose Winner With Evidence The tester scores outputs per case and aggregates: - expected content coverage - forbidden content violations - regex/format compliance - output length sanity Use the higher-scoring prompt as candidate baseline, then run regression suite. ### 3. Version Prompts ```bash # Add version python3 scripts/prompt_versioner.py add \ --name support_classifier \ --prompt-file prompts/support_v3.txt \ --author alice # Diff versions python3 scripts/prompt_versioner.py diff --name support_classifier --from-version 2 --to-version 3 # Changelog python3 scripts/prompt_versioner.py changelog --name support_classifier ``` ### 4. Regression Loop 1. Store baseline version. 2. Propose prompt edits. 3. Re-run A/B test. 4. Promote only if score and safety constraints improve. ## Script Interfaces - `python3 scripts/prompt_tester.py --help` - Reads prompts/cases from stdin or `--input` - Optional external runner command - Emits text or JSON metrics - `python3 scripts/prompt_versioner.py --help` - Manages prompt history (`add`, `list`, `diff`, `changelog`) - Stores metadata and content snapshots locally ## Pitfalls, Best Practices & Review Checklist **Avoid these mistakes:** 1. Picking prompts from single-case outputs — use a realistic, edge-case-rich test suite. 2. Changing prompt and model simultaneously — always isolate variables. 3. Missing `must_not_contain` (forbidden-content) checks in evaluation criteria. 4. Editing prompts without version metadata, author, or change rationale. 5. Skipping semantic diffs before deploying a new prompt version. 6. Optimizing one benchmark while harming edge cases — track the full suite. 7. Model swap without rerunning the baseline A/B suite. **Before promoting any prompt, confirm:** - [ ] Task intent is explicit and unambiguous. - [ ] Output schema/format is explicit. - [ ] Safety and exclusion constraints are explicit. - [ ] No contradictory instructions. - [ ] No unnecessary verbosity tokens. - [ ] A/B score improves and violation count stays at zero. ## References - [references/prompt-templates.md](references/prompt-templates.md) — 6 production marketing templates (ad copy, email sequence, social repurposing, landing sections, SEO meta, brand-voice rewrite) plus generic building blocks; each written to be graded by `prompt_tester.py` - [references/technique-guide.md](references/technique-guide.md) — technique-selection table for marketing tasks + the LLM-governance stack for marketing teams (claim discipline, disclosure rules, data boundaries, human-review gates) - [references/evaluation-rubric.md](references/evaluation-rubric.md) — mechanical scoring weights, acceptance gates, marketing quality dimensions, test-suite design, and eval anti-patterns - [README.md](README.md) ## Evaluation Design Each test case should define: - `input`: realistic production-like input - `expected_contains`: required markers/content - `forbidden_contains`: disallowed phrases or unsafe content - `expected_regex`: required structural patterns This enables deterministic grading across prompt variants. ## Versioning Policy - Use semantic prompt identifiers per feature (`support_classifier`, `ad_copy_shortform`). - Record author + change note for every revision. - Never overwrite historical versions. - Diff before promoting a new prompt to production. ## Rollout Strategy 1. Create baseline prompt version. 2. Propose candidate prompt. 3. Run A/B suite against same cases. 4. Promote only if winner improves average and keeps violation count at zero. 5. Track post-release feedback and feed new failure cases back into test suite.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.