ai-generated-test-review
Use this skill when reviewing AI-generated unit, functional, API, or end-to-end tests for false confidence, weak assertions, missing risks, or unsafe test behavior; triggers include AI-generated test review and functional test review.
Install
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-generated-test-review
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install naodeng-awesome-qa-skills@llmmart
git clone https://github.com/naodeng/awesome-qa-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole naodeng/awesome-qa-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
AI-Generated Test Review
Determine whether AI-generated tests prove real behavior instead of merely raising coverage or producing green builds.
When to Use
- Reviewing, changing, or accepting AI-generated unit, functional, API, or E2E tests.
- A test passes but its assertions, isolation, data, or coverage value is doubtful.
Workflow
- Inventory test types and files actually present in scope: unit, functional test, API, E2E, or a combination. Review only discovered or user-specified types.
- Establish the behavior, risk, and available requirement or implementation evidence. Mark missing evidence; do not invent findings.
- Read
prompts/review-test.mdand report actionable findings by severity, tied to a test, risk, and repair direction. - Load rules only for discovered layers: unit, functional test, API, or E2E. Load more than one only for cross-layer concerns; do not perform a global review just because multiple rule files exist.
- Separate merge-blocking faults from improvements; do not call an effective test defective because of style preference.
Domain Execution Essentials
- Evaluate assertions against the actual contract; mock, snapshot, or status-only syntax alone does not imply WEAK/FAKE.
- A boundary mock may replace a confirmed external dependency while real production logic executes. Mocking the subject itself and asserting its preset return offers no protection.
- A reviewed, independent snapshot of stable business structure can detect meaningful mutations. A snapshot that discards all business fields cannot. A status assertion can be STRONG when status fully expresses the supplied contract; persistence obligations require business postconditions.
- Support each classification with a concrete escaping/detected mutation and contract source. Retain confidence/gaps when evidence is missing, using the same bilingual rationale.
Core Constraints
- Judge observable behavior, failure signals, and risk coverage—not line coverage, test names, or mock-call counts alone.
- When a broken implementation could still pass, state the smallest break and the assertion needed to catch it.
- Never recommend deleting assertions, swallowing errors, loosening timeouts, or changing production behavior merely to obtain green tests.
- Do not expose credentials, personal data, or production write operations in tests, logs, or examples.
Progressive Disclosure
When calibrating domain outcomes, valid controls, or evidence gaps, read references/domain-decision-example.md for sanitized input, per-item oracles, and reproducible calculations.
Always read
prompts/review-test.mdbefore reviewing; its test-type routing precedes per-test review.For fake tests, excessive mocks, or ineffective assertions, read
references/fake-test-patterns.md.Read
references/unit-test-rules.md,references/functional-test-rules.md,references/api-test-rules.md, orreferences/e2e-test-rules.mdfor the applicable layer.Read related material under
examples/good/orexamples/bad/when an example is useful.
Pre-delivery Checklist
- States scope, evidence, and open questions
- Ranks findings and includes location, impact, and repair direction
- Reviews real assertions, negative paths, boundaries, isolation, and repeatability
- Does not present speculation as fact
Files (awesome-qa-skills)
-
agents
-
openai.yaml 396 B
Loading…
-
-
evals
-
cases
-
basic-success.yaml 769 B
Loading… -
domain-method-regression.yaml 1.6 KB
Loading… -
edge-incomplete-input.yaml 404 B
Loading… -
edge-risk-boundary.yaml 649 B
Loading… -
fake-mocked-subject.yaml 861 B
Loading… -
functional-route.yaml 856 B
Loading… -
strong-behavior.yaml 923 B
Loading…
-
-
fixtures
-
check-domain-decisions.sh 95 B
Loading… -
check_domain_decisions.py 3.7 KB
Loading… -
domain-oracle.json 154 B
Loading…
-
-
eval.yaml 458 B
Loading…
-
-
examples
-
bad
-
mock-only.md 290 B
Loading… -
mock-only.test.mjs 471 B · in bundle
-
-
good
-
behavior-first.md 256 B
Loading… -
behavior-first.test.mjs 805 B · in bundle
-
-
-
prompts
-
review-test.md 3.5 KB
Loading…
-
-
references
-
api-test-rules.md 628 B
Loading… -
domain-decision-example.md 1.7 KB
Loading… -
e2e-test-rules.md 657 B
Loading… -
fake-test-patterns.md 1.7 KB
Loading… -
functional-test-rules.md 698 B
Loading… -
unit-test-rules.md 688 B
Loading…
-
-
SKILL.md 3.9 KB
--- name: ai-generated-test-review description: Use this skill when reviewing AI-generated unit, functional, API, or end-to-end tests for false confidence, weak assertions, missing risks, or unsafe test behavior; triggers include AI-generated test review and functional test review. license: PolyForm-Noncommercial-1.0.0 metadata: version: "1.0.0" compatibility: "" author: "Awesome QA Skills contributors" language: "en" category: "testing-types" source: "https://github.com/naodeng/awesome-qa-skills" license-url: "https://polyformproject.org/licenses/noncommercial/1.0.0" --- # AI-Generated Test Review Determine whether AI-generated tests prove real behavior instead of merely raising coverage or producing green builds. ## When to Use - Reviewing, changing, or accepting AI-generated unit, functional, API, or E2E tests. - A test passes but its assertions, isolation, data, or coverage value is doubtful. ## Workflow 1. Inventory test types and files actually present in scope: unit, functional test, API, E2E, or a combination. Review only discovered or user-specified types. 2. Establish the behavior, risk, and available requirement or implementation evidence. Mark missing evidence; do not invent findings. 3. Read `prompts/review-test.md` and report actionable findings by severity, tied to a test, risk, and repair direction. 4. Load rules only for discovered layers: unit, functional test, API, or E2E. Load more than one only for cross-layer concerns; do not perform a global review just because multiple rule files exist. 5. Separate merge-blocking faults from improvements; do not call an effective test defective because of style preference. ## Domain Execution Essentials 1. Evaluate assertions against the actual contract; mock, snapshot, or status-only syntax alone does not imply WEAK/FAKE. 2. A boundary mock may replace a confirmed external dependency while real production logic executes. Mocking the subject itself and asserting its preset return offers no protection. 3. A reviewed, independent snapshot of stable business structure can detect meaningful mutations. A snapshot that discards all business fields cannot. A status assertion can be STRONG when status fully expresses the supplied contract; persistence obligations require business postconditions. 4. Support each classification with a concrete escaping/detected mutation and contract source. Retain confidence/gaps when evidence is missing, using the same bilingual rationale. ## Core Constraints - Judge observable behavior, failure signals, and risk coverage—not line coverage, test names, or mock-call counts alone. - When a broken implementation could still pass, state the smallest break and the assertion needed to catch it. - Never recommend deleting assertions, swallowing errors, loosening timeouts, or changing production behavior merely to obtain green tests. - Do not expose credentials, personal data, or production write operations in tests, logs, or examples. ## Progressive Disclosure - When calibrating domain outcomes, valid controls, or evidence gaps, read [references/domain-decision-example.md](references/domain-decision-example.md) for sanitized input, per-item oracles, and reproducible calculations. - Always read `prompts/review-test.md` before reviewing; its test-type routing precedes per-test review. - For fake tests, excessive mocks, or ineffective assertions, read `references/fake-test-patterns.md`. - Read `references/unit-test-rules.md`, `references/functional-test-rules.md`, `references/api-test-rules.md`, or `references/e2e-test-rules.md` for the applicable layer. - Read related material under `examples/good/` or `examples/bad/` when an example is useful. ## Pre-delivery Checklist - [ ] States scope, evidence, and open questions - [ ] Ranks findings and includes location, impact, and repair direction - [ ] Reviews real assertions, negative paths, boundaries, isolation, and repeatability - [ ] Does not present speculation as fact
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.