test-automation
Plan, write, and review automated tests following KATA (Komponent Action Test Architecture) on Playwright + TypeScript, or explain existing automated tests in a sealed read-only mode. Use when writing E2E or API/integration tests, creating Page or Api components, designing ATCs,
Install
npx skills add https://github.com/upex-galaxy/agentic-qa-boilerplate/tree/main/.agents/skills/test-automation
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install upex-galaxy-agentic-qa-boilerplate@llmmart
git clone https://github.com/upex-galaxy/agentic-qa-boilerplate.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole upex-galaxy/agentic-qa-boilerplate collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Forbidden invocations
NEVER invoke /sdd-* skills from this workflow. SDD is an optional
user-installed ceremony; this skill ships self-contained and does not chain
SDD under any condition. If you need to refactor KATA, fixtures, cli/,
scripts/, or api/schemas/ pipeline, exit this skill first and invoke
/framework-development — which itself runs Plan → Code → Verify → Archive
natively (no SDD required).
This boundary is mechanical, not advisory: scripts/lint-skills.ts rejects
any /sdd- mention outside this section. See:
.agents/skills/agentic-qa-core/references/skill-composition-strategy.md §4
(governs users who manually install SDD).
Test Automation — Plan, Code, Review
Produce KATA-compliant automated tests for an existing Playwright + TypeScript project. Three phases, always in this order: Plan → Code → Review. Never jump straight to code.
KATA (Komponent Action Test Architecture) rewires the usual Page Object pattern. If you write tests the "standard" way, most will be rejected at review. Load the relevant reference before writing code in that area.
Dependencies
Requires agentic-qa-core. Loads on demand:
agentic-qa-core/references/test-design-doctrine.md— MANDATORY before planning which ATCs to write from acceptance criteria. Governs the 1:N ATC derivation, the formal-technique triggers (incl. BVA, which KATA's EP-merge rule does NOT replace), and the floor-not-ceiling coverage model.agentic-qa-core/references/briefing-template.md,agentic-qa-core/references/dispatch-patterns.md,agentic-qa-core/references/orchestration-doctrine.md,agentic-qa-core/references/session-management.md,agentic-qa-core/references/preflight-gate.md,agentic-qa-core/references/adr-doctrine.md— cited inline by the sections that use them.
Compact Rules
Test-design doctrine (binding — full canon: agentic-qa-core/references/test-design-doctrine.md):
- "All ACs covered" is the FLOOR, not the success bar. The ATC set must also cover risk-beyond-AC: invalid/boundary inputs, auth/error paths, state transitions, and anomalies the AC is silent on.
- 1:N is the default: one AC maps to multiple ATCs. EP-merge collapses same-behavior inputs INSIDE one partition into a parameterized ATC — it must NEVER collapse across distinct partitions, boundaries, or states. BVA cases are required wherever a range/limit/length/date-window exists (EP alone misses off-by-one).
- Apply techniques by trigger: EP always; BVA on ranges/limits; State-Transition for stateful flows; Decision Table when 2+ conditions interact; Pairwise when 3+ combinable factors (log the reduction).
- Parametrize for artifact economy: same-behavior data variants → ONE parameterized
@atc(fixture / data-factory rows iterated by the test) per partition, NOT N ATCs; split only when action / outcome / state differs. (Canon: doctrine §"Part 2.5".) - An AC is the business assertion; an ATC is its concrete exploration (Precondition + Action + Assertions). Run the Test-Design Checklist before finalizing the plan.
Test-automation operational rules:
- Plan → Code → Review, always in order. Only automate
Candidateverdicts from/test-documentation. - Fixture selection: API-only →
{ api }(no browser); UI-only →{ ui }; hybrid →{ test }. - ATC = atomic mini-flow; NEVER calls another ATC. Reusable chains → a Steps module.
- Max 2 positional params (3+ → object param). Locators inline (extract only at 2+ uses). Imports via aliases (
@api/,@schemas/,@utils/) — no relative imports. - Public methods fail fast; utilities silent-fail (return null). Validate against
kata-manifest.jsonbefore adding components/ATCs (anti-duplication gate).
Read full SKILL.md when: writing KATA component code, choosing fixtures for a hybrid flow, or applying the Phase 3 review checklist.
Mode routing
Resolve mode before any readiness preflight or session workflow.
explain: selected by the legacybreak-down-testsalias or an explicit request to explain existing automated tests. Forward$ARGUMENTSunchanged, load onlyreferences/explain-tests.md, produce its read-only report, then stop. Do not create session state, run Plan -> Code -> Review, edit tests, regeneratekata-manifest.json, or call Jira/TMS.automate(default): all normal KATA planning, coding, and review triggers. Continue with the workflow below.
If the invocation could mean either explanation or implementation, ask which outcome is wanted. Never infer implementation from a read-only explanation request.
Subagent Dispatch Strategy
Orchestration & Session contracts: this skill follows
agentic-qa-core/references/orchestration-doctrine.md(mandatory subagent dispatch — main thread is command center) ANDagentic-qa-core/references/session-management.md(Phase 0 resume check, plan-first persistence at.session/<skill-slug>/<scope>/, archive on completion). Phase 0 (resume check) and Phase 1 (plan write) are NOT optional. The orchestrator also applies the per-stage Definition-of-Done gates inagentic-qa-core/references/stage-gates.md: verify a stage's DoD (planning stages include the Test-Design Checklist) BEFORE recording its progress checkpoint and advancing.
This skill is per-scope: <scope> = <JIRA-KEY> (ticket-driven / regression-driven) or <module-slug> (module-driven). Session state lives at .session/test-automation/<scope>/{plan.md, progress.md} per agentic-qa-core/references/session-management.md §3 + §9. The session plan.md is a thin INDEX that cites the canonical domain artifacts (spec.md, automation-plan.md, atc/*.md) under the Epic's test-specs/ tree (.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/) — domain content stays in the existing PBI tree, not duplicated.
This skill is compliant with the doctrine in AGENTS.md §"Orchestration Mode (Subagent Strategy)" and the session contract in .agents/skills/agentic-qa-core/references/session-management.md. Every dispatch follows the 7-component briefing format defined in .agents/skills/agentic-qa-core/references/briefing-template.md, and the pattern selected per phase matches the decision guide in .agents/skills/agentic-qa-core/references/dispatch-patterns.md. The Plan, Code, and Review phases each carry distinct context-isolation needs — Plan keeps KATA architectural reads out of the orchestrator, Code isolates multi-file edits, Review fans out three independent verifiers in parallel.
| Stage | Pattern | Subagent role |
|---|---|---|
Plan (spec.md + automation-plan.md) |
Single | one Plan subagent returns the two artifacts; protects orchestrator from KATA architectural reads |
| Code (writing E2E or API tests) | Sequential | one Code subagent per scope (module = 1 subagent per TC; ticket = 1 subagent total); edits-many-files inside isolates context |
Review — bun run test |
Parallel (sub-stage) | one Verifier subagent runs the test suite |
Review — bun run types:check |
Parallel (sub-stage) | one Verifier subagent runs typecheck |
Review — bun run lint:check |
Parallel (sub-stage) | one Verifier subagent runs lint |
| Review aggregation + merge/reject decision | Single | inline — orchestrator reads the 3 Verifier reports and decides |
- Code phase scope rule: each Code subagent edits multiple files in isolation, returns a list of changed files + a one-line summary per file. The orchestrator never reads the diffs — only the summary. If the user wants to see actual diffs, the orchestrator runs
git diffinline after the subagent returns. - On any Verifier failure: STOP, return the failing report verbatim to the user, do NOT auto-fix the test code, do NOT re-dispatch the Code phase without user approval. See
.agents/skills/agentic-qa-core/references/orchestration-doctrine.md. - MANDATORY context doc for Plan + Code briefings: include
kata-manifest.json(root) in the "Context docs" component (item 2 of the 7-component briefing). Without it the subagent will scantests/components/**directly, burn tokens, and risk proposing duplicates. See Critical Rule #12 inAGENTS.md.
Inputs — read these first, in this order
Canonical reading order for any AI starting cold on a test-automation workflow. Read in order; stop earlier when later inputs add no signal for the scope at hand.
kata-manifest.json(root) — authoritative registry of every Component (api[],ui[]) and every@atc('TICKET-ID')ID. Anti-duplication gate per Critical Rule #12 inAGENTS.md. MUST load before proposing any newPage,Api,Stepsmodule, or@atcID..agents/skills/test-automation/references/kata-architecture.md+.agents/skills/test-automation/references/typescript-patterns.md— full doctrine for KATA layers (TestContext / Base / Domain / Fixture), ATC identity, fixture selection, import-alias rules, params contracts.tests/components/— existing Api / Page / Steps shape on disk. Establishes naming, helper-vs-ATC split, fixture registration patterns to follow.The Story's
implementation-plan.md(dev plan) + the ATP under.context/PBI/epics/EPIC-<KEY>-<slug>/stories/STORY-<KEY>-<slug>/— Jira-synced, READ-ONLY caches. Jira is source of truth; NEVER hand-write these. Materialize viabun run jira:sync-issues get <STORY-KEY> --include-comments, then read the ENTIRE synced Story folder — every per-field.md(story.md,acceptance-criteria.md, scope, business rules, etc.) pluscomments.md— not just one field. Omitting ACs, scope, business rules, or comment context produces incomplete ATCs. The ATP read is modality-aware (resolve via.agents/project.yamltesting.tms_cli, same gate as/test-documentation§Phase 0):- Modality jira-native: ATP = Story field
{{jira.acceptance_test_plan}}→ syncedacceptance-test-plan.mdin the Story folder (from the samejira:sync-issues get <STORY-KEY> --include-comments). - Modality jira-xray: ATP = Test Plan issue
description→bun run jira:sync-issues get <ATP_KEY>→test-plans/ATP-<KEY>-<slug>.md; per-TC run results come from[TMS_TOOL](xray-cli), not the sync. Filename note: the acronym prefix comes from a conforming ladder title; a Plan or Execution whose title does not follow the grammar keeps the legacyTESTPLAN-/TESTEXEC-/RETESTEXEC-prefix.
The dev
implementation-plan.mdcarries the implementation approach + (when produced by/test-documentation) the per-TC candidate verdict and component mapping. Cite from the sessionplan.mdrather than duplicating. NOTE: this is the Story-folder devimplementation-plan.md— do NOT confuse it with the hand-authored automation plan (test-specs/<scope>/automation-plan.md) you write in Phase 1.- Modality jira-native: ATP = Story field
The Story's AC (acceptance criteria) — source of truth for scenarios that become ATCs. Read from the same synced
.mdfiles (acceptance-criteria.md/story.md) produced bybun run jira:sync-issues get <STORY-KEY> --include-comments. NEVER use[ISSUE_TRACKER_TOOL]viewfor these custom fields —viewreturnsnullforcustomfield_*. If a field is absent from the instance, the sync emits a pointer stub and the content lives in comments/description per.agents/jira-required.yamlfallback:. Resolve the issue key from the scope picker. TC note: a TC body = theTestissuedescription(synced both modalities viabun run jira:sync-issues get <TEST-KEY>); the Xray Gherkin / Test-Steps plugin field is NOT synced — it mirrors the description, so read the synced TC.mdfor Gherkin/steps.api/schemas/— OpenAPI-derived TypeScript types. Refresh viabun run api:syncif stale. Required for any Api component touching a new endpoint..env— credentials (LOCAL_USER_EMAIL,STAGING_USER_PASSWORD, etc.) read viaconfig.testUserfrom@variables. Never hardcode; never guess.agentic-qa-core/references/artifact-lifecycle.md— the TC status ladder this skill owns (candidate→in_automation→pull_request→automated), the unmapped-status fallback (§4), and the light stage verifier that closes Review (§5). Read BEFORE firing any transition.
Readiness Preflight Gate (MANDATORY — runs before Phase 0)
Full doctrine:
agentic-qa-core/references/preflight-gate.md. Runs FIRST, before the resume check and scope picker. Two laws: (1) args-as-answers — the scope, ticket key, and "API test" vs "E2E test" are provided args; ask only the gaps. (2) probe, don't assume. Surface gaps + REDs as ONEAskUserQuestionchecklist; self-fix with approval + explanation; STOP on any blocking RED. Note: this is distinct from the anti-duplication "Pre-flight checklist" inside Phase 1 (which cross-checkskata-manifest.jsonfor reuse) — this gate is about tools + env being ready to write and run code. Generic baseline (env resolution, test-user creds, secret/restart handling, the two laws, output contract) is inherited from the reference §3.1 — not repeated here. Below is only this skill's specific capability delta.
| Capability | Need | Why here |
|---|---|---|
| Framework adapted (artifacts present) | REQUIRED | Cannot write project ATCs against the generic Example* scaffolds the boilerplate ships. Probe the reference §4 ADAPTED signals; still generic → STOP and tell the user to run /project-discovery → /adapt-framework themselves. The gate NEVER auto-runs them. |
| Dev toolchain | REQUIRED | The Review gate runs bun run test / bun run types:check / bun run lint:check. Resolve them at t=0, not at Phase 3. bun install if a dep is missing. |
kata-manifest.json clean |
REQUIRED | Anti-duplication source of truth (Critical Rule #12). bun run kata:manifest:check clean before proposing components/ATCs; bun run kata:manifest if stale. |
| Active env + test-user creds | REQUIRED | Authored tests run live against <<ACTIVE_ENV>>. Env reachable + .env creds for the env (per role if multi-role). |
| Playwright browsers | REQUIRED | bunx playwright resolves + chromium installed (bun run pw:install). |
OpenAPI MCP (schema read-only) + api/schemas/ synced |
SCOPE — API/integration tests; needed at Phase 1 Plan too | Phase 1 explores endpoints (via the openapi MCP, schema-read-only) to design ATCs + classify test-data — plan-time, not just run-time. Api components consume OpenAPI-derived types (api/schemas/; refresh bun run api:sync); authenticated test-code calls use the Playwright API fixture (.auth/api-state.json from bun run api:login) — no API_TOKEN/MCP injection, no restart. |
| DBHub MCP | SCOPE — data setup/validation; needed at Phase 1 Plan too | Phase 1 explores the schema (via the dbhub MCP) to design data fixtures (Discover / Modify / Generate) — plan-time, not just run-time. dbhub answers a schema probe; DBHUB_* in .env. Unset → fill .env + RESTART. |
Issue-tracker ([ISSUE_TRACKER_TOOL]) |
SCOPE — ticket/regression-driven | ATP + AC reads via bun run jira:sync-issues; TMS modality for the ATP source. Pure module-driven from an existing spec may not need it. |
Surfaces (UI vs API vs both) follow the chosen planning scope + the ATCs Phase 1 designs — NEVER a user question (reference §5). After the gate clears (generic baseline + the surface tools the scope needs GREEN), continue to Phase 0 below.
Phase 0 — Resume check (MANDATORY, inline)
Before picking the planning scope, run the session resume contract from agentic-qa-core/references/session-management.md §4:
- Determine the prospective
<scope>from the invocation context (ticket key, regression-driven TC, or module slug — see "Pick the planning scope first" below). - Check
.session/test-automation/<scope>/progress.md. - If it does NOT exist → proceed to scope picker + Phase 1.
- If it DOES exist:
- Read
plan.md(thin index) + the tail ofprogress.md. - Read the cited canonical
spec.md/automation-plan.md/atc/*.mdunder.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/for the domain content. - Surface to the user: last completed phase (Plan / Code / Review) + next phase + open Review findings if any.
- Offer resume / restart / abort. On
restart, archive to.session/.archive/<YYYY-MM-DD>-test-automation-<scope>-aborted/before proceeding.
- Read
Phase 0 is inline (no subagent). It runs in <1 minute on a cold cache.
Pick the planning scope first
Every automation session starts by choosing one of three planning scopes. Pick once, then follow the Plan → Code → Review pipeline.
| Scope | Input | Output | Use when |
|---|---|---|---|
| Module-driven (Macro) | A module name + list of candidate TCs | One module spec + N ATC specs | Batch-automating an entire module (10+ tests). First pass on a new area. |
| Ticket-driven (Medium) | A single ticket/story ID with scenarios | One implementation plan for that ticket | Automating one user story end-to-end. Default for sprint work. |
| Regression-driven (Micro) | One specific TC (often after a bug fix) | One ATC implementation plan | Adding a single regression test after a fix. Smallest unit of work. |
When in doubt, ask the user which scope. Never assume "module" just because multiple TC IDs appear in the briefing.
These scopes consume the Candidate verdicts from /test-documentation (Stage 4) — only Candidate TCs reach automation; Manual / Deferred are terminal. The mapping from that skill's 4 documentation scopes: Module (Macro) ← module-driven, Ticket (Medium) ← ticket-driven, Regression-driven (Micro) ← bug-driven. Candidates from an ad-hoc / exploratory documentation session enter under whichever fits — a module batch, or regression-driven for a single TC.
Batch mode (fleet) — optional second executor
A batch (module-driven, or several ticket-driven scopes queued together) runs sequentially in one session by default: one Work Package at a time, Plan → Code → Review each. That is this skill's behaviour and it does not change.
A batch fleet — several persistent sessions working different Work Packages at once, coordinated by a conductor — is opt-in. Enter it only when the user asks for it, or when the batch holds 3+ Work Packages that touch disjoint modules. Everything below is a scoping rule; the launch and lifecycle transport lives in orca-orchestration/SKILL.md ([ORCHESTRATION_TOOL]), never here.
- Work Package (WP) = one delivery unit = one
test-specs/<ID>/spec (spec.md+automation-plan.md+atc/*.mdunder the Epic'stest-specs/tree). Sprint origin: theCandidateTCs of one Story. Discovery origin: a Tech Story. A batch is a list of WPs, never a list of files. - Partition by module, not by ticket. One worker owns every WP that touches a module's components; WPs sharing a module run sequentially inside that worker. NEVER two workers on the same module — same-module WPs share Pages / Apis / fixtures and collide in the files with the least merge tolerance.
- One worktree per worker. This skill writes code, and two sessions in one checkout contend on the git index even when their files are disjoint. Branch + PR per
git_strategy(sdet= one trunk, see.agents/skills/git-flow-master/references/sdet-integration-trunk.md). - The conductor regenerates
kata-manifest.jsonper integration (bun run kata:manifest). It is generated output — never hand-merged, never resolved as a text conflict. - The batch runs with or without an orchestration binary. The conductor always writes the launch file (one self-contained line per worker); with the binary those exact lines are launched for it, without it the human pastes them. Same payload either way, and nothing about the absence is reported to the user.
Full protocol — partition algorithm, collision table, conductor-only operations, integration order, per-worker brief: references/batch-fleet.md.
Workflow — Plan → Code → Review
Phase 1: Plan -> Phase 2: Code -> Phase 3: Review
(spec / plan) (component + test file) (KATA compliance)
| | |
.context/PBI/epics/ tests/components/** Review checklist
EPIC-<KEY>-<slug>/ tests/e2e/** or (pass/fail)
test-specs/<scope>/ tests/integration/**
spec.md
automation-plan.md
atc/*.md Register in fixture
Each phase has a gate. Do not start Code before the Plan is written and approved. Do not close out a ticket until Review passes.
Phase 1 — Plan
MUST-load before any planning: kata-manifest.json (root). It lists every Component and every ATC currently in the codebase. Use it to identify reuse, avoid duplicate Page/Api classes, and avoid minting an @atc('PROJ-XXX') ID that is already taken. This is enforced by Critical Rule #12 in AGENTS.md and by the husky pre-commit gate.
Pre-flight checklist (anti-duplication — run before writing the plan):
- Load
kata-manifest.json. Cross-check every proposed TC ID againstcomponents.api[].atcs[].idandcomponents.ui[].atcs[].id. If a match exists, the TC is already automated — re-scope or reuse. - Cross-check every proposed Component name against
components.api[].nameandcomponents.ui[].name. If a match exists, extend the existing class — do not create a new one. - If reuse opportunity exists (same flow already covered by a Steps method or ATC), adapt the plan to extend rather than rebuild.
Write the canonical domain plan file(s) for the chosen scope under the Epic's test-specs/ tree, .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/ (spec.md, automation-plan.md, atc/*.md). These are NON-Jira hand-authored files (committed to git). The automation automation-plan.md is distinct from the Story-folder dev implementation-plan.md (Jira-synced, read-only). The plan answers:
- Which scenarios from the ticket become tests, which become ATCs, which are shared preconditions (Steps)?
- Which components already exist (
tests/components/api/*Api.ts,tests/components/ui/*Page.ts) and which need to be created? - What test data is required? Classify by Discover / Modify / Generate (never assume data exists in staging).
- Which fixture will the test use --
{api},{ui}, or{test}? Do any shared preconditions call for a Steps class (tests/components/steps/*Steps.ts), instantiated directly in the test? - Which ATC IDs (from the TMS) map to which component methods?
Also write the session index plan.md at .session/test-automation/<scope>/plan.md per agentic-qa-core/references/session-management.md §6. This is a THIN INDEX — Goal, Inputs (cites the canonical artifacts above), Approach, Phase breakdown table, Risks, Verification checklist, Cross-references. It does NOT duplicate the domain content; it points to it.
Use the dispatch defined in §Subagent Dispatch Strategy: Single. Full briefing in references/planning-playbook.md §Plan dispatch.
Present the plan to the user. Wait for approval before coding. After approval, the orchestrator appends ## Phase 1 — Plan — <ts> with status: completed, artifacts_touched: [list of domain + session files], next: Phase 2 — Code to .session/test-automation/<scope>/progress.md.
Phase 2 — Code
Use the dispatch defined in §Subagent Dispatch Strategy: Sequential (one subagent per scope unit). The subagent loads references/e2e-patterns.md and references/api-patterns.md per scope.
Skills to load in every Code subagent (mandatory): /playwright-best-practices (community, project-installed) for upstream Playwright/TypeScript patterns — flaky-test fixes, POM vs fixtures, axe-core, auth/OAuth, fixtures lifecycle, perf budgets, i18n, component testing. Load alongside /test-automation (this skill, project-authored) — the two are complementary: KATA-specific rules (ATC identity, inline locators, fixture selection) come from here; generic Playwright craft comes from /playwright-best-practices. Add /playwright-cli only when the subagent also needs to drive a real browser session (snapshot/trace/record) during code-time exploration.
Open the TC lifecycle first. Before the first line of code, move every in-scope TC out of {{jira.status.test_case.candidate}}: [ISSUE_TRACKER_TOOL] Transition: {{jira.transition.test_case.start_automation}} → {{jira.status.test_case.in_automation}}. A TC left at candidate while its code is being written tells the team nobody picked it up. Unmapped slug → agentic-qa-core/references/artifact-lifecycle.md §4 fallback (ask, never skip silently).
Implement in this order:
- Types at top of component file (payloads, responses, domain DTOs).
- Component class extending
ApiBaseorUiBase. Helpers first (no decorator), ATCs second (@atc('TICKET-ID')). - Register the component in
tests/components/ApiFixture.tsorUiFixture.tsas appropriate. Steps classes (tests/components/steps/*Steps.ts) are NOT registered in any fixture — tests and setup files instantiate them directly (seeExampleSteps.ts). - Test file under
tests/e2e/{module}/ortests/integration/{module}/, using the correct fixture. - Run + verify, in this exact order -- do not skip steps:
bun run test <path/to/new.test.ts> # does it pass?
bun run types:check # tsc --noEmit, no errors
bun run lint:check # ESLint, no errors
If any step fails, fix before moving to Review.
- Signal it in the TMS — per TC, both signals, neither optional:
- Link: bind the automated test to the manual
Testit automates via thetest_automationlink type ({{jira.link_types.test_automation}}, outwardautomation test for— slug-resolved, never a literal; catalog:agentic-qa-core/references/traceability-linking.md§3). - Labels: ADD
automatedon theTestand REMOVEautomation-candidate— the two are mutually exclusive (test-documentation/SKILL.md§Labels). Flip them at the point the TC actually reaches AUTOMATED (see §Git & TMS handoff: themergedtransition after the suite PR lands onmainwith CI green), not on a merge into the integration trunk.
- Link: bind the automated test to the manual
[ISSUE_TRACKER_TOOL] Link work items: {AUTOMATED_TEST_KEY} -> {TC_KEY} type: {{jira.link_types.test_automation}}
[ISSUE_TRACKER_TOOL] Update work item: {TC_KEY} labels: + automated - automation-candidate
Progress checkpoint: after each Code subagent returns (per scope unit — one per TC for module-driven, one total for ticket-driven), the orchestrator appends a phase entry to .session/test-automation/<scope>/progress.md per agentic-qa-core/references/session-management.md §7. For module-driven scope, mid-batch resume reads the entries and skips already-coded ATCs.
AI-readable verification (optional, recommended)
For the test you just wrote, run Allure 3 in agent mode to get a markdown report you can read directly without parsing HTML:
bun allure:agent # runs `bunx allure agent -- bun test`
Allure 3 lives as a devDep — bunx allure resolves to the local node_modules/.bin/allure, no global install required. Use this when:
- The Code subagent needs to confirm the test actually exercised the expected ATC (the markdown summary lists each
@atc('TICKET-ID')block + its status). - You want a quick scope check before opening Phase 3 — Review.
- You are debugging a fixture/locator mismatch and want a structured failure read instead of a raw stack.
For human review of the same run, switch to:
bun allure:run # bunx allure run -- bun test (full HTML report)
bun allure:open # serve the last generated report locally
See regression-testing for CI / suite-level reporting (bun allure:generate, bun allure:watch).
Phase 3 — Review
Use the dispatch defined in §Subagent Dispatch Strategy: Parallel (3 simultaneous Verifiers). Full briefings in references/review-checklists.md §Parallel verification dispatch.
Run the review checklist on the new/modified files. Treat every failed item as a blocker. A clean review is the merge gate. See references/review-checklists.md for the full lists (E2E and API have overlapping but distinct checklists).
Required separate verifier — before merge, run /pr-review-lead or /judgment-day against the diff in a clean context (a fresh session/subagent with no memory of how the code was written). This is not opt-in and not limited to high-risk changes: Automation is the only stage whose autonomy reaches 3, and per agentic-qa-core/references/stage-gates.md it is the only stage with a mandatory separate verifier — more rope on the way in is paid for with a harder check on the way out. /judgment-day runs two blind judges in parallel against the diff and only approves when both agree (see .agents/skills/judgment-day/SKILL.md); /pr-review-lead runs a QA-lead-style review grounded in KATA doctrine. Pick whichever fits the change; skipping this step is a Review DoD failure, not a shortcut.
Light stage verifier (closes the Automation stage) — run the eight-line template in agentic-qa-core/references/artifact-lifecycle.md §5. Stage-specific lines:
[ ] Every in-scope TC at {{jira.status.test_case.in_automation}} or beyond (start_automation fired)
[ ] TCs bound to their automated test via the {{jira.link_types.test_automation}} link
[ ] Labels flipped only at the status they belong to (+automated / -automation-candidate
at `merged`, never at trunk merge)
[ ] No TC left at {{jira.status.test_case.candidate}} with code already written for it
[ ] Any unmapped slug went through the §4 fallback (asked), never a silent skip
Progress checkpoint + Archive: after Phase 3 returns ACCEPT (all 3 Verifiers exit 0), the orchestrator appends ## Phase 3 — Review — <ts> with status: completed, next: stop to .session/test-automation/<scope>/progress.md, then runs Archive per agentic-qa-core/references/session-management.md §8: moves .session/test-automation/<scope>/ to .session/.archive/<YYYY-MM-DD>-test-automation-<scope>/ (two-file dir preserved) and calls mem_session_summary including the archive path. On REJECT, archive does NOT run — the working directory stays for debug.
Git & TMS handoff (sdet integration-trunk suites)
This skill stops at a clean local review. It does not create branches, push, or open PRs — that is /git-flow-master's job. When the repo's git strategy is sdet (the standing mode for chained test-automation suites), each ticket flows through the per-ticket loop in .agents/skills/git-flow-master/references/sdet-integration-trunk.md:
The Phase 3 ACCEPT gate (3 Verifiers green:
test/types:check/lint:check) is the skill's local validation gate. Undersdetit must pass on both thelocalandstagingenvironments before push — re-run the suite against each (active_envper.agents/project.yaml). The Verifiers are local-only; Sanity CI on the branch is owned by/git-flow-master+/regression-testing, never by this skill.After ACCEPT, surface the explicit handoff — "Local gate green. Ready for
/git-flow-master: cuttest/{KEY}-{slug}from the integration trunk, push, Sanity-CI, PR into the trunk, merge--no-ff." Do not auto-invoke git operations.Append the Git Ledger line to the suite's
progress.mdafter each branch action (orchestrator-written, append-only) so a resuming session knows how the trunk was left: trunk name + SHA, last ticket merged, pending tickets, sync-gate / final-PR state. Schema in../agentic-qa-core/references/session-management.md§7 "The Git Ledger"; what-to-write detail in.agents/skills/git-flow-master/references/sdet-integration-trunk.md§Resume.TC lifecycle anchors to the ticket-branch PR, not the final
trunk → mainPR. The full ladder this skill owns (canon:agentic-qa-core/references/artifact-lifecycle.md§1, Test row):Moment Transition Status after Phase 2 — Code opens {{jira.transition.test_case.start_automation}}{{jira.status.test_case.in_automation}}ticket PR opens into the trunk {{jira.transition.test_case.create_pr}}{{jira.status.test_case.pull_request}}final suite PR merges to main, CI green there{{jira.transition.test_case.merged}}{{jira.status.test_case.automated}}Merging into the trunk is NOT
automated. Execute transitions via/test-documentation+[ISSUE_TRACKER_TOOL]; resolve every slug through.agents/jira-workflows.json, and on an unmapped slug run theartifact-lifecycle.md§4 fallback instead of skipping.
Fixture selection (inline — load-bearing every invocation)
The fixture you pick determines whether a browser opens. Wrong fixture = slow API tests or missing UI context.
| Test type | Fixture | Browser opens? | Use when |
|---|---|---|---|
| API only (integration) | { api } |
No (lazy) | Pure API testing. No UI needed. |
| UI only | { ui } |
Yes | UI-focused testing. No backend setup via API. |
| Hybrid (UI + API setup) | { test } |
Yes | Setup data via API, drive flow via UI, verify via API. |
Rules:
- Only three fixtures exist:
{ api },{ ui },{ test }(seetests/components/TestFixture.ts). Reusable precondition chains (3+ ATCs repeated across 3+ files) go in a Steps class undertests/components/steps/— instantiated directly in the test, never exposed as a fixture. - Integration tests (
tests/integration/**) almost always use{ api }. - E2E tests (
tests/e2e/**) use{ ui }if no API setup needed, otherwise{ test }. - Never request
{ ui }for a test that never interacts with the UI -- it opens a browser for nothing.
Gotchas (inline — the most common rejection reasons)
- ATC = complete test case, not a single click.
clickLoginButton()is not an ATC.loginWithValidCredentials(credentials)is. If the method is one-line-wrappingpage.click(), delete it. - TC Identity Rule: Precondition + Action = 1 TC. All expected results from the same precondition and same action are assertions of the same TC, not separate TCs. Do not split a TC across panels, endpoints, or UI sections.
- Equivalence Partitioning. Same expected output = one parameterized ATC. Three ATCs all returning HTTP 401 for invalid login are wrong -- merge into one
loginWithInvalidCredentials(payload). - ATCs do not call ATCs. ATCs are atomic. For reusable chains, use the Steps module (
tests/components/steps/*Steps.ts). Steps are NOT decorated with@atc. - Locators inline. No
locators/*.tsfiles. Put the selector in the ATC. If the same locator is used in 2+ ATCs of the same component, extract it to aprivate readonlyarrow function in the class -- not to a separate file. - Helpers vs ATCs. A read-only GET is a helper (no
@atc, optionally@step). An action that changes state is an ATC (@atc('TICKET-ID')). A GET inside an ATC that verifies the action succeeded is fine -- but the GET alone is not an ATC. - Fixed assertions go inside ATCs. Status code, required fields, URL redirect checks. Test-level assertions (flow outcomes) go in the test file.
- Max 2 positional parameters. 3+ parameters must use an object parameter (
fn(args: Args)). Named object parameters beat positional lists for maintainability and autocomplete. - Import aliases are mandatory.
@api/,@ui/,@utils/,@variables,@TestContext,@schemas/. No relative imports (../../../). Lint will reject them. - No hardcoded waits. Never
page.waitForTimeout(3000). Wait for a specific condition:waitForSelector,waitForResponse,waitForLoadState('networkidle'), ordata-loaded="true"attributes. - No retries by default.
retries: 0inplaywright.config.ts. If a test passes on retry, it is flaky, not passing. Investigate. - Credentials from
.env, never hardcoded.LOCAL_USER_EMAIL/STAGING_USER_EMAILand their passwords. Read viaconfig.testUserfrom@variables. - Each test generates its own data. No shared state between tests. Use
TestContext.generateUserData()or faker helpers for unique values. - Ticket ID prefix in every
test(). Format:test('TICKET-ID: should {behavior} when {condition}', ...). Thedescribeblock may also include the ticket ID when the file is tied to a single ticket. - One component per file, one file per feature. Components follow
{Resource}Api.tsor{Page}Page.ts. Test files follow{verb}{Feature}.test.ts(e.g.,applyDiscount.test.ts, neverdiscount.test.ts). - Don't propose components or ATCs without consulting the manifest.
kata-manifest.jsonis the registry. Skipping it produces (a) duplicate Pages — proposingLoginPagewhenLoginPage.tsalready exists; (b) duplicate ATC IDs — minting@atc('PROJ-90')twice; (c) missed reuse — creatinggetBookingByIdwhenBookingsApi.getByIdalready does it. Always start the Plan phase by loading the manifest. The husky pre-commit gate enforces freshness; Critical Rule #12 inAGENTS.mdenforces consultation. - Cross-cutting test-architecture decisions become ADRs, not plan-buried prose. When Plan or Code reveals a decision that is architectural AND hard to reverse — a fixture lifecycle reused across 3+ ATCs or 2+ tickets, a test-data-isolation contract, an auth-in-tests change, a flake-retry-policy shift, a Page-Object-vs-Screenplay move — promote it from
planning-playbook.md§2 "Architecture Decisions" to a standalone.context/ADR/ADR-NNNN-<slug>.mdand leave aSee ADR-NNNNbacklink. Ticket-local choices stay in the plan. ADRs are append-only: supersede, never rewrite. Seeagentic-qa-core/references/adr-doctrine.md. - Session-footer contract (mandatory at close). The final phase is not done until the two chat-facing blocks from
../agentic-qa-core/references/session-footer-contract.mdare printed: (1) consolidated screenshot list — repo-relative paths, verified on disk, bug annotations first — plus in-flow surfacing of every capture's path the instant it lands; (2) Session Footer listing skills/MCPs/CLIs actually used + testing levels touched, with explicit "none" entries for expected-but-untouched levels. Framing for this skill: authoring. Multi-subagent sessions: each stage report carries the five footer fields (skills_loaded,mcps_used,clis_used,testing_levels_touched,screenshots_captured); the orchestrator compiles the footer ONCE at close. Chat only — never in a Jira comment or ATR body.
Minimal templates (inline — small, load-bearing)
KATA component signatures
// API component — Layer 3
export class UsersApi extends ApiBase {
constructor(options: TestContextOptions) { super(options); }
// Helper (read-only)
@step
async getUserById(id: string): Promise<[APIResponse, UserResponse]> {
return this.apiGET<UserResponse>(`/users/${id}`);
}
// ATC (state-changing)
@atc('TICKET-ID')
async createUserSuccessfully(payload: UserPayload): Promise<[APIResponse, UserResponse, UserPayload]> {
const [response, body, sent] = await this.apiPOST<UserResponse, UserPayload>('/users', payload);
expect(response.status()).toBe(201);
expect(body.id).toBeDefined();
return [response, body, sent];
}
}
// UI component — Layer 3
export class LoginPage extends UiBase {
constructor(options: TestContextOptions) { super(options); }
@atc('TICKET-ID')
async loginWithValidCredentials(data: LoginData): Promise<void> {
await this.page.goto('/login');
await this.page.locator('#email').fill(data.email);
await this.page.locator('#password').fill(data.password);
await this.page.locator('button[type="submit"]').click();
await expect(this.page).toHaveURL(/.*dashboard.*/);
}
}
Test file skeleton
import { test, expect } from '@TestFixture';
import usersData from '@data/fixtures/users.json';
test.describe('TICKET-ID: Validate discount codes', () => {
test('TICKET-ID: should apply percentage discount when code is valid', async ({ api }) => {
const order = await api.orders.createOrderSuccessfully(orderData);
const totals = await api.orders.getTotals({ orderId: order.id });
expect(totals.finalAmount).toBe(totals.baseAmount - totals.discountAmount);
});
});
Fixture registration (excerpt)
// tests/components/UiFixture.ts
export class UiFixture extends TestContext {
readonly login: LoginPage;
readonly checkout: CheckoutPage;
constructor(options: TestContextOptions) {
super(options);
this.login = new LoginPage(options);
this.checkout = new CheckoutPage(options);
}
}
Quality gates (must pass before merge)
| Gate | Command | Must be |
|---|---|---|
| Tests pass | bun run test {path} |
All green, zero retries used |
| Types | bun run types:check |
No errors |
| Lint | bun run lint:check |
No errors |
| Fixture registered | visual | Component is in ApiFixture / UiFixture (Steps classes are instantiated directly, never registered) |
| ATC IDs linked | visual | Every @atc('X') matches a real TMS test case ID |
| Naming | visual | Files PascalCase for components, camelCase verb for test files |
| Session footer | chat | Session footer + consolidated screenshot list printed in chat per session-footer-contract (never in a Jira comment) |
Anti-patterns — NEVER do these
T1. NEVER auto-generate tests for TCs that /test-documentation flagged as Deferred or Manual — only Candidate (to_be_automated) verdicts proceed to automation. Skipping the ROI verdict produces flaky, low-value suites.
T2. NEVER skip the Plan phase. Even for a "simple" regression test, write spec.md / automation-plan.md (or the per-ATC plan under .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/) BEFORE writing any test code. Plan → Code → Review is non-negotiable.
T3. NEVER collapse the KATA layers (TestContext / ApiBase + UiBase / Domain Api+Page+Steps / Fixture). Full doctrine in references/kata-architecture.md. Tests that flatten layers are rejected at Review.
T4. NEVER call one ATC from inside another ATC. ATCs are atomic mini-flows. Reusable chains live in the Steps module (tests/components/steps/*Steps.ts), which is NOT decorated with @atc.
T5. NEVER use relative imports (../../../). This repo uses path aliases (@api/, @ui/, @schemas/, @utils/, @TestContext, @variables, @TestFixture). Lint rejects relative paths.
T6. NEVER hardcode credentials. Read from .env via config.testUser from @variables (LOCAL_USER_* / STAGING_USER_*). Never inline a password, token, or API key in a spec, fixture, or test data file.
T7. NEVER hardcode customfield_NNNNN in spec files, test data, or test config. Resolve Jira fields via {{jira.<slug>}} against .agents/jira-fields.json + .agents/jira-required.yaml so test code survives workspace rotations.
T8. NEVER mix test code and product code in the same PR. Test PRs follow the test/* branch convention with title format {type}({ISSUE-KEY}): {description} — see .agents/skills/git-flow-master/references/pr-test-automation.md. Under the sdet strategy, adjacent non-test work never rides a test/* ticket branch either — it goes on a Plus Branch (docs/*/chore/*/fix/* → integration trunk). See .agents/skills/git-flow-master/references/sdet-integration-trunk.md.
Which reference to read
Not every invocation needs every reference. Load the specific file when the task matches.
- KATA architecture, fixture selection, ATC rules, Steps module mechanics →
references/kata-architecture.md - TypeScript conventions (params, imports, types, errors, DRY by layer) →
references/typescript-patterns.md - Naming, tagging, folder structure, anti-patterns, quality gates →
references/automation-standards.md - Writing a Playwright
Pagecomponent, locator strategy, data-testid rules, UI waits →references/e2e-patterns.md - Writing an
Apicomponent, OpenAPI type facades, HTTP helper usage, schema imports →references/api-patterns.md - Designing test data (Discover → Modify → Generate), fixtures JSON, faker →
references/test-data-management.md @atc/@stepdecorators, NDJSON results, TMS sync mechanics →references/atc-tracing.md- Writing the Plan (module / ticket / ATC scopes and templates) →
references/planning-playbook.md - Running a batch across several parallel sessions (partition by module, conductor duties, integration order) →
references/batch-fleet.md - Running the review checklist (E2E or API) →
references/review-checklists.md - Configuring Playwright, CI integration, projects, sharding →
references/ci-integration.md - Session resume contract, plan.md/progress.md schemas, archive policy, Engram per-phase checkpoint →
../agentic-qa-core/references/session-management.md(Phase 0 + Phase 1 + Archive of this skill)
Tool resolution: use [AUTOMATION_TOOL] for browser work (Playwright CLI or MCP — load /playwright-cli when available), [API_TOOL] for OpenAPI exploration, [DB_TOOL] for verifying test data in the database, [TMS_TOOL] for TMS sync (load /xray-cli when available), [ISSUE_TRACKER_TOOL] for ticket work. Split the issue-tracker access by operation: detailed reads of a Story (ACs, ATP, dev implementation-plan, custom fields) → bun run jira:sync-issues get <KEY> --include-comments (or jql "<query>") then read the synced .md — NEVER acli workitem view for custom fields; writes (comment automated-test status back to the Story, transitions) → /acli; trivial summary/status/key-list lookups → /acli workitem view/search is fine. See agentic-qa-core/references/acli-integration.md §"Reads vs writes". Resolve tags via the project's AGENTS.md Tool Resolution table.
Quick reference
# Planning outputs (hand-authored, NON-Jira; Epic-level test-specs/)
# .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/spec.md
# .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/automation-plan.md
# .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/atc/*.md
# (the Story-folder implementation-plan.md is the Jira-synced DEV plan — read-only input, not written here)
# Code locations
# tests/components/api/{Resource}Api.ts
# tests/components/ui/{Page}Page.ts
# tests/components/steps/{Domain}Steps.ts
# tests/integration/{module}/{verbFeature}.test.ts
# tests/e2e/{module}/{verbFeature}.test.ts
# Local validation loop
bun run test <path>
bun run types:check
bun run lint:check
bun run kata:manifest # regenerate registry if components/ATCs changed
git add kata-manifest.json # stage so the freshness gate passes
bun run kata:manifest:check # confirm gate would pass (husky runs this on commit)
# Env + TMS sync
cp .env.example .env # populate test credentials
bun run api:sync # regenerate OpenAPI schema types
Files (agentic-qa-boilerplate)
-
evals
-
evals.json 2.7 KB
{ "evals": [ { "name": "should-trigger-write-e2e-test", "prompt": "Write an E2E test for the checkout flow. The user adds a product to the cart, goes to checkout, and confirms the purchase.", "expected_behavior": "Activates test-automation. Plans ATCs first (CheckoutPage + supporting Api components), writes implementation plan, then codes the KATA component and test file using the correct fixture. Runs tests, type-check, and lint before marking done.", "category": "positive" }, { "name": "should-trigger-create-api-component", "prompt": "I need an API component for the endpoint POST /api/orders. Follow the KATA pattern.", "expected_behavior": "Activates test-automation. Creates OrdersApi in Layer 3 extending ApiBase, uses tuple returns [APIResponse, Body, Payload], decorates state-changing methods with @atc('TICKET-ID'), registers in ApiFixture, and writes an integration test using the { api } fixture.", "category": "positive" }, { "name": "should-trigger-review-test-code", "prompt": "Review the test I wrote at tests/e2e/processCheckout.test.ts. Verify it follows KATA.", "expected_behavior": "Activates test-automation. Runs the E2E review checklist: layer placement, ATC naming, inline locators, no ATC-calling-ATC, fixture choice, ticket-ID prefixes, no hardcoded waits, import aliases. Reports each failed item as a blocker.", "category": "positive" }, { "name": "should-not-trigger-run-regression-suite", "prompt": "Trigger the regression suite and tell me whether we can ship to production.", "expected_behavior": "Does NOT activate test-automation. Should route to regression-testing, which handles gh workflow run, Allure analysis, failure classification, and GO/NO-GO verdicts.", "category": "negative" }, { "name": "should-not-trigger-document-test-case", "prompt": "Document these test cases in Jira as Xray Tests and compute the ROI for which ones to automate.", "expected_behavior": "Does NOT activate test-automation. Should route to test-documentation, which handles TMS artifact creation, ROI scoring, and automation-candidate tagging.", "category": "negative" }, { "name": "should-not-trigger-onboard-project", "prompt": "Set up this new project in the testing boilerplate and generate the business-data-map and api-architecture context files.", "expected_behavior": "Does NOT activate test-automation. Should route to project-discovery, which runs the 4-phase discovery pipeline and generates the .context/ artifacts; KATA adaptation follows via the `/adapt-framework` command.", "category": "negative" } ] }
-
-
references
-
api-patterns.md 22.6 KB
# API Patterns — Components, OpenAPI Types, Status Matrix > **Subagent context**: when the test-automation Code phase is dispatched (Sequential, see SKILL.md §Subagent Dispatch Strategy), this file is part of the subagent's "Context docs" briefing component. How to write a KATA `Api` component and integration test. Load when building a new API component, importing OpenAPI types, choosing ATCs for success/error status codes, or handling auth headers. KATA layer mechanics (tuple returns, `ApiBase.apiGET/POST/...`, ATC atomicity, fixture selection) live in `kata-architecture.md` and `automation-standards.md` — this reference only covers API-specific mechanics. --- ## 1. Where API code lives ``` api/ openapi.json # downloaded spec (gitignored) openapi-types.ts # auto-generated types (committed) .openapi-config.json # sync metadata (gitignored) schemas/ # type facade files (see §4) index.ts # barrel re-export auth.types.ts # one file per domain orders.types.ts tests/components/api/ ApiBase.ts # Layer 2 — HTTP helpers, auth, Allure attach AuthApi.ts # Layer 3 — one file per resource OrdersApi.ts tests/components/ApiFixture.ts # registers every Api component tests/integration/{module}/{verbResource}.test.ts ``` Integration tests live under `tests/integration/**` and use the `{ api }` fixture (no browser opens). --- ## 2. API component skeleton ```typescript import type { APIResponse } from '@playwright/test'; import type { TestContextOptions } from '@TestContext'; import { expect } from '@playwright/test'; import { ApiBase } from '@api/ApiBase'; import { atc, step } from '@utils/decorators'; // Import typed payloads/responses from the facade — NEVER from '@openapi' import type { Order, CreateOrderRequest, CreateOrderResponse, OrderErrorResponse, } from '@schemas/orders.types'; export class OrdersApi extends ApiBase { private readonly baseEndpoint = '/api/v1/orders'; constructor(options: TestContextOptions) { super(options); } @step async getOrderById(id: string): Promise<[APIResponse, Order]> { return this.apiGET<Order>(`${this.baseEndpoint}/${id}`); } @atc('TICKET-ID') async createOrderSuccessfully( payload: CreateOrderRequest, ): Promise<[APIResponse, CreateOrderResponse, CreateOrderRequest]> { const [response, body, sent] = await this.apiPOST<CreateOrderResponse, CreateOrderRequest>( this.baseEndpoint, payload, ); expect(response.status()).toBe(201); expect(body.id).toBeDefined(); expect(body.customerId).toBe(payload.customerId); return [response, body, sent]; } } ``` Method order: base endpoint → constructor → helpers (`@step`, no `@atc`) → success ATCs → error ATCs. Max 15–20 ATCs per file; split by resource. --- ## 3. ApiBase — HTTP helpers ApiBase wraps `APIRequestContext` with typed tuple returns and automatic Allure attachment. Never call `request.fetch` directly from a Layer-3 component. | Method | Return | Purpose | |--------|--------|---------| | `apiGET<T>(path, opts?)` | `[APIResponse, T]` | Read | | `apiPOST<T, P>(path, payload, opts?)` | `[APIResponse, T, P]` | Create | | `apiPUT<T, P>(path, payload, opts?)` | `[APIResponse, T, P]` | Full update | | `apiPATCH<T, P>(path, payload, opts?)` | `[APIResponse, T, P]` | Partial update | | `apiDELETE<T>(path, opts?)` | `[APIResponse, T]` | Delete | ### Tuple pattern GET/DELETE return 2-tuples (`response`, `body`). Mutation verbs return 3-tuples (`response`, `body`, `sentPayload`) so downstream assertions can compare what went in against what came back. ```typescript const [response, body] = await this.apiGET<UserResponse>('/users/1'); const [resp, body, sent] = await this.apiPOST<UserResponse, CreateUserPayload>('/users', payload); ``` ### Request options ```typescript interface RequestOptions { headers?: Record<string, string>; params?: Record<string, string>; timeout?: number; } const [response, body] = await this.apiGET<SearchResults>('/search', { params: { q: 'test', limit: '10' }, headers: { 'X-Custom-Header': 'value' }, timeout: 30000, }); ``` ### Query strings for list endpoints ```typescript const qs = new URLSearchParams(); if (params?.page) qs.set('page', String(params.page)); if (params?.limit) qs.set('limit', String(params.limit)); const endpoint = qs.toString() ? `${this.baseEndpoint}?${qs}` : this.baseEndpoint; const [response, body] = await this.apiGET<OrderListResponse>(endpoint); ``` Use `URLSearchParams` for safety; never string-concatenate values directly. --- ## 4. OpenAPI type facades The single most important rule when OpenAPI is available: **only facade files import from `@openapi`**. Components import from `@schemas/{domain}.types`. This keeps the generated file cohesive and lets you evolve type names without rewriting every component. ### Generation flow ``` ┌──────────────────────┐ ┌───────────────────────────────────┐ │ Backend (running) │ │ Test Repo (KATA) │ │ /swagger/v1/ │── fetch ──> │ api/openapi.json (gitignored)│ │ swagger.json │ │ ↓ openapi-typescript │ └──────────────────────┘ │ api/openapi-types.ts (committed) │ │ ↓ re-exported │ │ api/schemas/{domain}.types.ts │ │ ↓ imported │ │ tests/components/api/{Domain}Api │ └───────────────────────────────────┘ ``` **Prerequisite**: The backend must be running and exposing its Swagger/OpenAPI spec. The URL is configured in `api/.openapi-config.json` or passed via `--url`. If the backend is not running, `api:sync` will fail with `Connection refused`. **What gets committed**: Only `api/openapi-types.ts` is committed. `api/openapi.json` and `api/.openapi-config.json` are gitignored — they are local cache artifacts. ### Sync commands | Command | Effect | |---------|--------| | `bun run api:sync` | Download spec + regenerate `openapi-types.ts` (default) | | `bun run api:sync --url <url>` | Pull from a specific URL | | `bun run api:sync --no-types` | Download only, skip type generation | | `bun run api:sync --help` | Help | Run `bun run api:sync` before every automation session that touches new endpoints. Commit the regenerated `openapi-types.ts`. `openapi.json` and `.openapi-config.json` stay gitignored. ### Facade file template Each `api/schemas/{domain}.types.ts` has up to three sections: ```typescript import type { components, paths } from '@openapi'; // ── Schema types — domain models from components.schemas ── export type Order = components['schemas']['OrderListModel']; export type Product = components['schemas']['ProductModel']; // ── Endpoint types — POST /api/orders ── type CreateOrderPath = paths['/api/orders']['post']; export type CreateOrderRequest = CreateOrderPath['requestBody']['content']['application/json']; export type CreateOrderResponse = CreateOrderPath['responses']['201']['content']['application/json']; // ── Endpoint types — GET /api/orders/{id} ── type GetOrderPath = paths['/api/orders/{id}']['get']; export type GetOrderParams = GetOrderPath['parameters']['path']; export type GetOrderResponse = GetOrderPath['responses']['200']['content']['application/json']; // ── Custom types — not in the spec ── export interface OrderErrorResponse { error: string; statusCode?: number; message?: string; details?: Record<string, string[]>; } ``` ### Section rules | Section | Source | Use when | |---------|--------|----------| | Schema types | `components['schemas']` | Domain entities / DTOs | | Endpoint types | `paths[...][method]` | Request/response per endpoint | | Custom types | Plain `interface` | Error shapes, test helpers, anything not in the spec | Use a private helper `type XPath = paths[...][method]` so request/response/params extractions stay readable. ### Import map | Type origin | Location | Import from | |-------------|----------|-------------| | OpenAPI schema | `api/schemas/{domain}.types.ts` | `@schemas/{domain}.types` | | OpenAPI endpoint | `api/schemas/{domain}.types.ts` | `@schemas/{domain}.types` | | Not in spec, domain-specific | `api/schemas/{domain}.types.ts` § Custom | `@schemas/{domain}.types` | | Cross-domain shorthand | — | `@schemas` (barrel) | | Test data shapes (DataFactory) | `tests/data/types.ts` | `@data/types` | ```typescript // inside an Api component import type { LoginPayload, TokenResponse } from '@schemas/auth.types'; // inside a cross-domain test file import type { LoginPayload } from '@schemas/auth.types'; import type { Order } from '@schemas/orders.types'; // or via the barrel import type { LoginPayload, Order } from '@schemas'; ``` ### Creating a new facade 1. Copy `api/schemas/example.types.ts` to `api/schemas/{domain}.types.ts`. 2. Replace placeholder schema names with real ones from `api/openapi-types.ts`. 3. Add `export type * from './{domain}.types'` to `api/schemas/index.ts`. 4. Import from `@schemas/{domain}.types` in the component. ### Type navigation (reading the generated file) The generated `openapi-types.ts` exposes two top-level types: `components` (schema models) and `paths` (endpoint signatures). Navigate them as follows: ```typescript import type { components, paths } from '@openapi'; // only in facade files // Schema models (DTOs, entities) type Order = components['schemas']['OrderListModel']; // Endpoint request body type CreateOrderBody = paths['/api/orders']['post']['requestBody']['content']['application/json']; // Endpoint response body type GetOrdersResponse = paths['/api/orders']['get']['responses']['200']['content']['application/json']; // Path parameters type OrderPathParams = paths['/api/orders/{id}']['get']['parameters']['path']; // Query parameters (when the spec defines them) type OrderQueryParams = paths['/api/orders']['get']['parameters']['query']; ``` Use these patterns inside facade files only. Components never drill into `paths` or `components` directly. ### OpenAPI integration best practices 1. **Sync before writing tests** — run `bun run api:sync` to get the latest types before any automation session touching new endpoints. 2. **Commit `openapi-types.ts`** — the generated types file is committed so CI and other team members have them without running the backend. 3. **One facade per domain** — matches the API component it serves (`auth.types.ts` serves `AuthApi.ts`, `orders.types.ts` serves `OrdersApi.ts`). 4. **Only facades import `@openapi`** — components never import directly from the generated file. 5. **Re-export via barrel** — every facade must be re-exported from `api/schemas/index.ts` for cross-domain imports. --- ## 5. Status-code matrix — one ATC per expected outcome Every HTTP outcome is a separate ATC because the expected status AND the response shape differ. Same status + different payload family = one parameterised ATC. | Outcome | Status | ATC name pattern | Response type | |---------|--------|------------------|---------------| | Created | 201 | `create{Resource}Successfully` | `{Resource}Response` | | OK (read) | 200 | `get{Resource}Successfully` | `{Resource}Response` or `{Resource}ListResponse` | | OK (update) | 200 | `update{Resource}Successfully` | `{Resource}Response` | | No Content / OK (delete) | 204 or 200 | `delete{Resource}Successfully` | `void` | | Validation error | 400 | `create{Resource}WithInvalid{Field}` | `ApiErrorResponse` | | Unauthenticated | 401 | `get{Resources}Unauthorized` | `ApiErrorResponse` | | Forbidden | 403 | `delete{Resource}Forbidden` | `ApiErrorResponse` | | Not found | 404 | `get{Resource}WithNonExistentId` / `get{Resource}NotFound` | `ApiErrorResponse` | | Conflict | 409 | `create{Resource}WithDuplicate{Field}` | `ApiErrorResponse` | ### Success — 201 Created ```typescript @atc('TICKET-ID') async createOrderSuccessfully( payload: CreateOrderRequest, ): Promise<[APIResponse, CreateOrderResponse, CreateOrderRequest]> { const [response, body, sent] = await this.apiPOST<CreateOrderResponse, CreateOrderRequest>( this.baseEndpoint, payload, ); expect(response.status()).toBe(201); expect(body.id).toBeDefined(); expect(body.customerId).toBe(payload.customerId); return [response, body, sent]; } ``` ### Success — 200 OK (list with pagination) ```typescript @atc('TICKET-ID') async getAllOrdersSuccessfully( params?: { page?: number; limit?: number }, ): Promise<[APIResponse, OrderListResponse]> { const qs = new URLSearchParams(); if (params?.page) qs.set('page', String(params.page)); if (params?.limit) qs.set('limit', String(params.limit)); const endpoint = qs.toString() ? `${this.baseEndpoint}?${qs}` : this.baseEndpoint; const [response, body] = await this.apiGET<OrderListResponse>(endpoint); expect(response.status()).toBe(200); expect(Array.isArray(body.data)).toBe(true); expect(body.pagination).toBeDefined(); expect(body.pagination.page).toBeGreaterThanOrEqual(1); return [response, body]; } ``` ### Delete — accept 200 or 204 ```typescript @atc('TICKET-ID') async deleteOrderSuccessfully(id: string): Promise<[APIResponse, void]> { const [response] = await this.apiDELETE(`${this.baseEndpoint}/${id}`); expect([200, 204]).toContain(response.status()); return [response, undefined]; } ``` ### Error — 400 Validation ```typescript @atc('TICKET-ID') async createOrderWithInvalidPayload( payload: Partial<CreateOrderRequest>, ): Promise<[APIResponse, OrderErrorResponse, Partial<CreateOrderRequest>]> { const [response, body] = await this.apiPOST<OrderErrorResponse, Partial<CreateOrderRequest>>( this.baseEndpoint, payload, ); expect(response.status()).toBe(400); expect(body.error).toBeDefined(); return [response, body, payload]; } ``` ### Error — 401 Unauthorized (explicitly without auth) ```typescript @atc('TICKET-ID') async getOrdersUnauthorized(): Promise<[APIResponse, OrderErrorResponse]> { // Call api.clearAuthToken() in the test before calling this ATC const [response, body] = await this.apiGET<OrderErrorResponse>(this.baseEndpoint); expect(response.status()).toBe(401); expect(body.error).toBeDefined(); return [response, body]; } ``` ### Error — 404 Not Found ```typescript @atc('TICKET-ID') async getOrderWithNonExistentId(id: string): Promise<[APIResponse, OrderErrorResponse]> { const [response, body] = await this.apiGET<OrderErrorResponse>(`${this.baseEndpoint}/${id}`); expect(response.status()).toBe(404); expect(body.error).toBeDefined(); return [response, body]; } ``` Equivalence partitioning applies: three ATCs that all return 401 for different invalid logins collapse into one parameterised `loginWithInvalidCredentials(payload)`. See `kata-architecture.md` §6 Rule 3. --- ## 6. Authentication & token lifecycle ### Automatic token propagation A successful `loginSuccessfully` stores the token on `ApiBase`. Subsequent calls carry `Authorization: Bearer ...` automatically. ```typescript test('TICKET-ID: should make authenticated call', async ({ api }) => { await api.auth.signInSuccessfully({ email, password }); // stores token const [, orders] = await api.orders.getOrdersSuccessfully({ customerId: 123 }); }); ``` ### Manual token management ```typescript api.setAuthToken('eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...'); // set manually api.clearAuthToken(); // drop before 401 test ``` ### ApiFixture propagation pattern When a new component is added to `ApiFixture`, auth mutators must forward to it so `setAuthToken` / `clearAuthToken` affect every registered component. ```typescript export class ApiFixture extends ApiBase { readonly auth: AuthApi; readonly orders: OrdersApi; constructor(options: TestContextOptions) { super(options); this.auth = new AuthApi(options); this.orders = new OrdersApi(options); } // NOTE: there is no `setRequestContext` to override. The request context // arrives through the constructor (`new OrdersApi(options)`) and `ApiBase` // exposes it as a `get request()` getter. Only the two TOKEN methods need // forwarding, because a token is set AFTER construction and every component // holds its own copy. override setAuthToken(token: string) { super.setAuthToken(token); this.auth.setAuthToken(token); this.orders.setAuthToken(token); } override clearAuthToken() { super.clearAuthToken(); this.auth.clearAuthToken(); this.orders.clearAuthToken(); } } ``` Forgetting to wire a new component into `setAuthToken` is the most common cause of surprise 401s in API suites. --- ## 7. Custom error shape When the OpenAPI spec under-specifies errors (common), define a local `ApiErrorResponse` in the domain facade's Custom-types section. ```typescript export interface ApiErrorResponse { error: string; message?: string; statusCode: number; details?: Record<string, string[]>; } ``` Error ATCs parameterise the response type as `ApiErrorResponse` (not the success response) so the caller gets the right autocomplete: ```typescript const [response, body] = await this.apiGET<ApiErrorResponse>(`${this.baseEndpoint}/${id}`); ``` --- ## 8. Integration test file template ```typescript import { test, expect } from '@TestFixture'; import type { CreateOrderRequest } from '@schemas/orders.types'; test.describe('TICKET-ID: Validate orders API', () => { let customer: CustomerCandidate | null; test.beforeAll(async ({ api }) => { // DISCOVER (see test-data-management.md) — no assertions await api.auth.signInSuccessfully({ email, password }); customer = await api.customers.findAvailableCustomer(); }); test('TICKET-ID: should create order successfully @critical', async ({ api }) => { if (!customer) return test.skip(true, 'No available customer'); const payload = api.data.createOrder({ customerId: customer.id }); const [response, body, sent] = await api.orders.createOrderSuccessfully(payload); // Test-level assertions beyond the fixed ones in the ATC expect(body.customerId).toBe(sent.customerId); expect(body.status).toBe('pending'); }); test('TICKET-ID: should reject order with invalid payload', async ({ api }) => { if (!customer) return test.skip(true, 'No available customer'); const invalid = api.data.createOrder({ customerId: customer.id, quantity: -1 }); const [, error] = await api.orders.createOrderWithInvalidPayload(invalid); expect(error.error).toBeDefined(); }); test('TICKET-ID: should return 401 without auth', async ({ api }) => { api.clearAuthToken(); await api.orders.getOrdersUnauthorized(); }); }); ``` Rules applied above (all mandatory): - `{ api }` fixture for API-only tests — no browser opens. - Ticket ID prefix on describe and test. - `beforeAll` discovers state without assertions; each test guards with `test.skip()`. - `api.data.*` generates data; never hardcoded. - Type imports come from `@schemas/orders.types`, never from `@openapi`. - Auth-dependent tests clear the token before running. --- ## 9. Chaining ATCs at the test level ATCs are atomic. The test orchestrates them — one ATC never calls another. ```typescript test('TICKET-ID: should create order and appear in list', async ({ api }) => { await api.auth.signInSuccessfully({ email, password }); const [, order] = await api.orders.createOrderSuccessfully(orderData); const [, list] = await api.orders.getAllOrdersSuccessfully({ page: 1, limit: 50 }); expect(list.data.some(o => o.id === order.id)).toBe(true); }); ``` For preconditions repeated across 3+ tests in 3+ files, extract into a `Steps` module — not into an ATC that calls another ATC. See `kata-architecture.md` §8. --- ## 10. Running API tests ```bash # All integration tests bun run test:integration # One file bun run test tests/integration/auth/authenticateUser.test.ts # By tag bun run test --grep @smoke bun run test --grep @critical # Debug bun run test:integration --debug # Allure report bun run allure:generate # Regenerate types from a fresh OpenAPI spec (run before writing new API ATCs) bun run api:sync ``` --- ## 11. Allure attachments `ApiBase` attaches the full request/response pair to the Allure report on every HTTP call. No manual attachment needed. Masking of `password` / `token` / `secret` parameters happens in the `@atc` / `@step` decorators. Keep those parameter names canonical so masking works out of the box. --- ## 12. Troubleshooting — OpenAPI sync | Symptom | Likely cause | Fix | |---------|--------------|-----| | `Connection refused` during `api:sync` | Backend not running | Start backend on the URL in `api/.openapi-config.json` (or pass `--url`) | | Types not regenerating | Invalid `openapi.json` | Validate JSON; delete `api/openapi.json`; re-run sync | | Import error for `@api/*` or `@schemas/*` | Missing path alias | Check `tsconfig.json` `paths`; see `typescript-patterns.md` | | Type mismatch between spec and runtime | Stale generated file | `bun run api:sync` to re-pull and regenerate | | Missing endpoint in facade | Forgot step 3 when adding a domain | Re-export in `api/schemas/index.ts` | --- ## 13. Implementation checklist (API coding phase) Before leaving the coding phase: - [ ] Component extends `ApiBase`, constructor takes `TestContextOptions` and calls `super`. - [ ] `baseEndpoint` constant at the top of the class. - [ ] Helpers (`@step`) above ATCs (`@atc`). - [ ] Types imported from `@schemas/{domain}.types`; no direct `@openapi` imports in the component. - [ ] Tuple returns: 2-tuple for GET/DELETE, 3-tuple for POST/PUT/PATCH. - [ ] Every ATC contains at least one status-code assertion plus one shape assertion. - [ ] Error ATCs use `ApiErrorResponse` as the body type, not the success response. - [ ] Max 2 positional parameters; 3+ collapse to an object param. - [ ] `ApiFixture` registered the new component AND forwarded BOTH `setAuthToken` and `clearAuthToken` to it. There is no `setRequestContext`: the request context arrives through the constructor. - [ ] Test file under `tests/integration/{module}/`, name follows `{verb}{Resource}.test.ts`. - [ ] `test` imported from `@TestFixture`. - [ ] Ticket ID prefix in describe/test. - [ ] Data via `api.data.*`, never hardcoded. - [ ] 401 tests call `api.clearAuthToken()` first. - [ ] `bun run api:sync` executed if new endpoints were added. - [ ] `bun run test <file>` / `bun run types:check` / `bun run lint:check` all clean. -
atc-tracing.md 18.1 KB
# ATC Tracing and TMS Sync How KATA captures ATC execution results, persists them across worker processes, aggregates them into a final report, and syncs the outcomes back to the TMS. Load when debugging why results do not appear, extending the reporter chain, wiring TMS sync in CI, or adding a new TMS provider. Assumed knowledge: `@atc` / `@step` purpose, fixture selection, and parameter masking are covered in `kata-architecture.md`. This file focuses on the plumbing — NDJSON schema, reporter lifecycle, manifest output, TMS sync payloads. --- ## 1. Pipeline at a glance ``` Test workers (N) Coordinator (1) Teardown (1) ----------------- --------------- ------------ @atc/@step ──► test.step() KataReporter.onEnd() global.teardown │ │ │ ├─► KataReporter (terminal) ├─► read NDJSON lines ├─► read .atc_partial.ndjson ├─► allure-playwright (auto) ├─► aggregate by testId └─► show ATC Coverage summary └─► storeResult() → NDJSON ├─► write atc_results.json └─► delete NDJSON ── process exits ── │ └─► bun run test:sync → syncResults() ``` The teardown project finishes BEFORE `onEnd()`, so it reads the NDJSON partial file, not the aggregate — and it cannot sync, because the file the sync reads does not exist yet. The TMS write-back is a separate command run after the Playwright process exits. Key invariants: - Playwright spawns each project in its own worker; in-memory state does not survive. Durable result capture must go to disk. - The reporter `onEnd()` runs in the coordinator process, so the aggregate file is the single source of truth for everything downstream (teardown summary, TMS sync, dashboards). - Global teardown is a separate worker process. It reads the aggregated JSON — not the NDJSON — because `onEnd()` has already converted it. --- ## 2. NDJSON partial file ### 2.1 Location and format ``` reports/.atc_partial.ndjson ``` Each `@atc` invocation appends exactly one line. NDJSON (newline-delimited JSON) is used for safe concurrent appends: `appendFileSync` of a single line is atomic on POSIX filesystems. One line per ATC execution: ```json {"testId":"PROJ-101","methodName":"authenticateSuccessfully","className":"AuthApi","status":"PASS","error":null,"executedAt":"2026-03-27T19:34:58.470Z","duration":1835,"softFail":false} ``` ### 2.2 Line schema | Field | Type | Description | |-------|------|-------------| | `testId` | `string` | Ticket key passed to `@atc('...')` — e.g. `PROJ-101`. Must match the TMS issue key exactly. | | `methodName` | `string` | Decorated method name — useful when multiple ATCs share a ticket. | | `className` | `string` | Component class where the ATC lives (e.g. `AuthApi`). Disambiguates duplicates. | | `status` | `'PASS' \| 'FAIL' \| 'SKIP'` | Outcome of this invocation. | | `error` | `string \| null` | Error message if the ATC threw or an assertion failed. | | `executedAt` | `string` (ISO 8601) | UTC timestamp of the call. | | `duration` | `number` (ms) | Wall-clock time of the decorated method. | | `softFail` | `boolean` | Whether the `@atc({ softFail: true })` option was in effect. | ### 2.3 Lifecycle rules | Phase | What happens | Where | |-------|--------------|-------| | Test run start | File does not exist | — | | First `@atc` call | `storeResult()` creates the file and appends | any worker | | Subsequent calls | `appendFileSync` adds one line | any worker | | `KataReporter.onEnd()` | Reads every line, aggregates by `testId`, writes the final JSON | coordinator | | End of `onEnd()` | File is deleted — must not leak between runs | coordinator | Never read or edit the NDJSON outside the reporter. A teardown that reads it directly will double-count or race with the cleanup. --- ## 3. Aggregated result file ### 3.1 Location ``` reports/atc_results.json ``` Written by `KataReporter.onEnd()` in the coordinator process. Persists after the run — downstream tooling (teardown, TMS sync, dashboards) reads from here. ### 3.2 Schema ```json { "generatedAt": "2026-03-27T19:35:03.356Z", "summary": { "total": 2, "passed": 1, "failed": 1, "skipped": 0, "executions": 5, "testIds": ["PROJ-101", "PROJ-202"] }, "results": { "PROJ-101": [ { "testId": "PROJ-101", "methodName": "authenticateSuccessfully", "className": "AuthApi", "status": "PASS", "error": null, "executedAt": "2026-03-27T19:34:58.470Z", "duration": 1835, "softFail": false } ], "PROJ-202": [ { "testId": "PROJ-202", "methodName": "createOrder", "className": "OrdersApi", "status": "FAIL", "error": "Expected 201, received 500", "executedAt": "...", "duration": 342, "softFail": false } ] } } ``` ### 3.3 Summary field semantics | Field | Meaning | |-------|---------| | `total` | Count of **unique ATC IDs** that executed at least once — equal to `testIds.length`. | | `passed` | Unique ATCs where **every** execution passed. | | `failed` | Unique ATCs where **any** execution failed. Conservative rule — one bad run poisons the ATC. | | `skipped` | Unique ATCs where **every** execution was `SKIP`. | | `executions` | Raw count of NDJSON lines — sum of invocations across all tests. | | `testIds` | Sorted list of unique ATC IDs. Useful for CI diffs. | The split between `total` (unique ATCs) and `executions` (raw invocations) matters: an ATC used as a precondition in 10 tests will show `executions: 10` but contributes `1` to `total`. --- ## 4. ATCs are reusable — final status logic An ATC can be invoked from many places in a single run: ``` PROJ-101 authenticateSuccessfully ├── tests/setup/api-auth.setup.ts — once during setup ├── tests/integration/auth/login.test.ts — invoked as a precondition ├── tests/integration/orders/create.test.ts — invoked as a precondition └── tests/e2e/dashboard/home.test.ts — invoked via its LoginPage variant ``` When computing the status to send to the TMS: ```typescript const finalStatus = executions.every(e => e.status === 'PASS') ? 'PASSED' : 'FAILED'; ``` Rules and rationale: | Executions | Final status | Why | |------------|--------------|-----| | all PASS | `PASSED` | ATC is reliable across every context it was used in. | | any FAIL | `FAILED` | One broken context is a real failure — do not hide it behind a passing majority. | | only FAIL | `FAILED` | Straightforward. | | none | not synced | The ATC was not touched this run. Nothing to report. | A parameterised ATC that varies by input is still one ATC for TMS purposes — the `testId` is the unit, not the parameter set. --- ## 5. Sensitive parameter masking Both decorators mask argument values whose keys are in the `SENSITIVE_KEYS` set in `tests/utils/decorators.ts` (canonical keys: `password`, `token`, `secret`, `authorization`, `access_token`). See `api-patterns.md` for the canonical parameter-name rules that make masking work by default. To mask additional keys, add them to that set. --- ## 6. KataReporter output Both decorators wrap execution in `test.step()`. KataReporter subscribes to `onStepBegin` / `onStepEnd` / `onTestEnd` and prints a tree view: ``` 🧪 Running Test [3/6] => UPEX-100: should be able to re-authenticate ---- ✓ ATC [PROJ-101]: authenticateSuccessfully({ email: "user@test.com", password: "***" }) ---- API OK: 200 - https://app.example.com/api/auth/login ---- ✓ getCurrentUser() ---- API OK: 200 - https://app.example.com/api/auth/me ---- step passed ✅ [195ms] ---- step passed ✅ [820ms] ✅ [PROJ-101] authenticateSuccessfully - PASS (825ms) ---- 🔎 Test Output: ✅ PASSED ``` Line sources: | Line | Emitter | |------|---------| | `---- ✓ ATC [ID]: method(args)` | KataReporter `onStepBegin` | | `---- ✓ helper()` | KataReporter `onStepBegin` (nested) | | `---- API OK: 200 - url` | `ApiBase` — HTTP log | | `---- step passed ✅ [Nms]` | KataReporter `onStepEnd` | | `✅ [ID] method - PASS (Nms)` | `@atc` console.log | | `---- 🔎 Test Output: ✅ PASSED` | KataReporter `onTestEnd` | Filter criterion: KataReporter only prints steps where `step.category === 'test.step'`. Lifecycle hooks and `expect` steps are ignored. --- ## 7. Global teardown summary Before the reporter fires, the teardown prints a stand-alone summary from `reports/.atc_partial.ndjson`: ``` ============================================================ KATA Architecture - Global Teardown ============================================================ ATC Coverage: 1 unique ATC tracked (2 total executions) ✅ Passed: 1 | ❌ Failed: 0 | ⏭️ Skipped: 0 [SKIP] TMS sync is OFF — these results were NOT written back to the TMS. Set AUTO_SYNC=true to enable it, then run `bun run test:sync` after the suite. ============================================================ 📊 ATC Report generated: reports/atc_results.json ``` The sync does not run here. With `AUTO_SYNC=true` the teardown only says so; the write-back happens when `bun run test:sync` runs after the process exits. --- ## 8. TMS sync ### 8.1 Trigger A separate command, never an in-process call. `reports/atc_results.json` is written by `KataReporter.onEnd()`, which fires after every project — the teardown included — so anything that syncs from inside the run reads the previous run's file (local) or none at all (CI). ```bash AUTO_SYNC=true bun run test # writes reports/atc_results.json on exit bun run test:sync # reads it and writes the results back ``` The suite workflows under `.github/workflows/` run the second line as a `Sync Results to TMS` step gated on `AUTO_SYNC == 'true'`, right after the test step. `syncResults()` itself still honours the `AUTO_SYNC` gate, and in CI it exits non-zero when the report file is missing rather than returning silently. ### 8.2 Provider routing ``` syncResults() ├── provider = 'xray' → syncToXray() │ ├── POST /authenticate (bearer token) │ └── POST /import/execution (batch, all ATCs in one payload) │ ├── provider = 'jira' → syncToJiraDirect() │ ├── For each testId: │ │ ├── PUT /issue/{testId} (update Test Status custom field) │ │ └── POST /issue/{testId}/comment (execution details) │ └── Return { success, failed } │ └── provider = 'none' → skip ``` ### 8.3 Environment variables Required for either provider: ```env AUTO_SYNC=true TMS_PROVIDER=xray # 'xray' | 'jira' | 'none' ``` Xray Cloud: ```env XRAY_CLIENT_ID=... XRAY_CLIENT_SECRET=... XRAY_PROJECT_KEY=PROJ STP_EXECUTION_KEY=PROJ-456 # Xray only — the STR Test Execution the run writes back to ``` `STP_EXECUTION_KEY` decides WHERE results land. Despite the name it must hold the key of the **STR Test Execution** linked to the sprint STP, **never the STP's own key** — `tests/utils/jiraSync.ts` reads the target's issue type and refuses a Test Plan outright. Unset → every run mints a NEW, unparented Execution instead of appending to the STR. Jira Direct: ```env # NOTE: the Atlassian site HOST is not a .env variable. It lives in # .agents/project.yaml -> issue_tracker.atlassian_url (`bun run agents:setup`). ATLASSIAN_EMAIL=email@company.com ATLASSIAN_API_TOKEN=... JIRA_TEST_STATUS_FIELD={{jira.test_status}} # resolved at runtime against .agents/jira-fields.json (regenerate via `bun run jira:sync-fields --force`) ``` ### 8.4 Comment body sent to the TMS For each ATC, the provider writes a comment shaped like: ``` KATA ATC: authenticateSuccessfully Executions: 5 Duration: 577ms Last run: 2026-03-27T19:35:01.866Z Error: Expected 200, received 401 ``` The `Executions` count is the total invocations; the other fields come from the **last execution**. `Error` is omitted when the final status is `PASSED`. ### 8.5 Provider comparison | Aspect | Xray Cloud | Jira Direct | |--------|-----------|-------------| | Issue type used | `Test` (Xray) | any (`Task`, `Story`, ...) — custom field `Test Status` drives state | | API surface | Xray REST | Generic Jira REST v3 | | Batch upload | Yes — single `/import/execution` | No — one `PUT` + one `POST` per ATC | | Test plans / executions | Native | Not supported | | Cost | Paid Marketplace app | Free | Pick Jira Direct for small teams or MVPs; upgrade to Xray when reporting needs grow. ### 8.6 Test ID format ``` UPEX-123 e.g. DEMO-456 ``` Rules: - Must match the TMS issue key exactly (case-sensitive). - Issue must exist before sync — sync does not create tickets. - Any Jira issue type is acceptable for Jira Direct; Xray expects `Test` type specifically. --- ## 9. kata:manifest — static registry of what exists `bun run kata:manifest` is independent of test execution. It scans `tests/components/**` and emits `kata-manifest.json` describing every component and every `@atc(...)` call it finds. Use it to answer "what ATCs exist?" without running the suite or scanning source files manually. ### 9.1 Commands ```bash bun run kata:manifest # write kata-manifest.json at project root bun run kata:manifest:watch # regenerate on file change bun run kata:manifest --stdout # write to stdout instead of a file bun run kata:manifest:check # CI-grade freshness check; exits 1 if kata-manifest.json is stale. # Used by .husky/pre-commit when staged files touch # tests/components/, scripts/kata-manifest.ts, or kata-manifest.json. ``` Scan roots (hard-coded): ``` tests/components/api/**/*.ts tests/components/ui/**/*.ts tests/components/steps/**/*.ts ``` Excluded files: `ApiBase.ts`, `UiBase.ts`, `TestContext.ts`, `TestFixture.ts`, `ApiFixture.ts`, `UiFixture.ts`, `index.ts`. Extraction pattern: `@atc\s*\(\s*['"]([^'"]+)['"]` — a literal string key is required. Template literals and computed IDs are not picked up. ### 9.2 Output schema ```json { "version": "1.0", "generatedAt": "2026-04-14T09:12:00.000Z", "components": { "api": [ { "name": "AuthApi", "file": "AuthApi.ts", "relativePath": "tests/components/api/AuthApi.ts", "atcs": [ { "id": "PROJ-101", "method": "authenticateSuccessfully", "line": 42 }, { "id": "PROJ-102", "method": "authenticateWithInvalidCredentials", "line": 67 } ] } ], "ui": [ /* same shape */ ] }, "steps": [ { "name": "ExampleSteps", "file": "ExampleSteps.ts", "relativePath": "tests/components/steps/ExampleSteps.ts", "methods": ["authenticateUser", "createTestResource", "navigateAsAuthenticatedUser"] } ], "summary": { "totalComponents": 12, "totalATCs": 57, "apiComponents": 7, "uiComponents": 5, "stepsModules": 1 } } ``` Preconditions list method names only — Steps are not `@atc`-decorated and carry no TMS IDs. ### 9.3 When to run it | Situation | Why | |-----------|-----| | New AI session | Feed the manifest as context instead of scanning the tree | | CI precondition | Assert `totalATCs` did not drop unexpectedly (guard against accidental deletion) | | Coverage comparison | Diff manifest vs TMS ticket list to find ATCs not yet linked | | Planning phase | Check which components already exist before proposing new ones | The manifest is a static registry — it says what **exists in code**. `atc_results.json` says what **ran in the last test run**. They complement each other but are not interchangeable. --- ## 10. Debugging guide | Symptom | Likely cause | Fix | |---------|--------------|-----| | `Total: 0` in teardown summary | NDJSON never written — no `@atc` decorators applied, or only `@step` | Check decorators are present on state-changing methods | | ATC runs but nothing in terminal | Method not wrapped in `test.step()`, or reporter not in the chain | Verify `KataReporter` is listed in `playwright.config.ts` reporters; confirm `decorators.ts` imports `test` from `@playwright/test` | | Sensitive value visible in step title | Key not in `SENSITIVE_KEYS` | Add the key to `SENSITIVE_KEYS` in `tests/utils/decorators.ts` | | Sync reports "No ATC results to sync" | `atc_results.json` empty | Run tests first; check KataReporter ran `generateAtcReport()`; confirm reporter config | | Sync fails with 401 | Bad TMS credentials | Verify `XRAY_CLIENT_*` or `ATLASSIAN_API_TOKEN`; check expiry and permissions | | Sync fails with "Test key not found" | TMS issue missing or mistyped | Create the issue in the TMS first; check case-sensitive exact match | | Custom-field error (Jira Direct) | Wrong `JIRA_TEST_STATUS_FIELD` ID | `[ISSUE_TRACKER_TOOL] list_fields()` (load `/acli`) and grep for the field; or re-run `bun run jira:sync-fields --force` | | Sync is slow | Jira Direct = one request per ATC | Batch via Xray if budget allows; move sync off the local loop — run only in CI on `main` | --- ## 11. Files that own this pipeline | File | Responsibility | |------|----------------| | `tests/utils/decorators.ts` | `@atc`, `@step`, `formatArgs`, `storeResult` (NDJSON writer), `SENSITIVE_KEYS` | | `tests/KataReporter.ts` | Terminal tree output; `generateAtcReport()` in `onEnd()` (NDJSON → JSON); NDJSON cleanup | | `tests/teardown/global.teardown.ts` | Reads `.atc_partial.ndjson`, prints the ATC Coverage summary, states whether the write-back is on. Does NOT sync | | `tests/utils/jiraSync.ts` | `syncToXray()`, `syncToJiraDirect()`, provider router | | `playwright.config.ts` | Reporter chain (KataReporter must be registered), `global-teardown` PROJECT (wired via `teardown:` on the `global-setup` project, not a `globalTeardown` hook) | | `config/variables.ts` | `config.tms.*` — reads the env vars listed in §8.3 | | `scripts/kata-manifest.ts` | Static scanner — produces `kata-manifest.json` | | `reports/.atc_partial.ndjson` | Ephemeral per-run capture (deleted in `onEnd()`) | | `reports/atc_results.json` | Persistent aggregated result — input to `bun run test:sync` (written too late for the teardown) | | `kata-manifest.json` | Static registry of components and ATCs in source | -
automation-standards.md 32.7 KB
# Automation Standards Canonical conventions for naming, tagging, folder layout, data management, anti-patterns, and quality gates in KATA. Load when the standards need verification — before opening a PR, when reviewing test code, or when structuring a new module. Architectural and code-style rules live in `kata-architecture.md` and `typescript-patterns.md`. --- ## 1. Naming Conventions ### Components | Type | Class name | File name | |------|-----------|-----------| | Test Context | `TestContext` | `TestContext.ts` | | API Base | `ApiBase` | `ApiBase.ts` | | UI Base | `UiBase` | `UiBase.ts` | | API Component | `{Resource}Api` | `UsersApi.ts`, `OrdersApi.ts` | | UI Component | `{Page}Page` | `LoginPage.ts`, `CheckoutPage.ts` | | Steps | `{Domain}Steps` | `AuthSteps.ts`, `CheckoutSteps.ts` | | Fixture | `{Type}Fixture` | `ApiFixture.ts`, `UiFixture.ts`, `TestFixture.ts` | File naming: PascalCase, matches class name exactly. All code in English (components, ATCs, variables, comments). Documentation may be Spanish or English per team preference; skills and context docs are English-only. ### ATCs (Acceptance Test Cases) Pattern: `{verb}{Resource}{Scenario}` — always camelCase, always English. | Scenario | Suffix | Example | |----------|--------|---------| | Success | `Successfully` / `WithValidCredentials` | `signInSuccessfully`, `loginWithValidCredentials` | | Invalid input | `WithInvalid{X}` | `signInWithInvalidCredentials` | | Not found | `WithNonExistent{X}` | `getUserWithNonExistentId` | | Expired | `WithExpired{X}` | `loginWithExpiredToken` | | Forbidden | `WithRestricted{X}` / `WithInsufficient{X}` | `loadDashboardWithRestrictedRole` | Rules: - Name must describe **what action** is performed and **what outcome** is expected. - Must be a complete test case, not a single interaction. - `clickLoginButton()`, `fillEmailSuccessfully()`, `submitFormSuccessfully()` are **wrong** — they describe interactions, not test cases. ### Test Files | Type | Pattern | Example | |------|---------|---------| | E2E | `{verb}{Feature}.test.ts` | `processCheckout.test.ts`, `createSignup.test.ts` | | Integration | `{verb}{Resource}.test.ts` | `authenticateUser.test.ts`, `createOrder.test.ts` | | Utility | `{util}.test.ts` | `decorators.test.ts` | Rules: - File-name verb describes the **user action** (apply, create, submit, refresh), not a test verb like `verify`, `check`, or `test`. - One file = one feature. A feature may have multiple `describe` blocks (tickets) that touch the same functionality. - One ticket = one `describe`. Never split a ticket across files. - Multiple tickets can coexist in one file when they test different aspects of the same feature. ### Test Hierarchy The four-level hierarchy maps to how work is organized: ``` Module (directory) -> tests/e2e/orders/ Feature (file) -> applyDiscount.test.ts Ticket (describe) -> 'TICKET-ID: Apply Discount Code' Scenario (test) -> 'TICKET-ID: should apply percentage discount when code is valid' ``` | Level | Maps to | Naming | Example | |-------|---------|--------|---------| | Directory | Module / product area | kebab-case | `orders/`, `products/`, `auth/` | | File | Feature / functional area | `{verb}{Feature}.test.ts` (camelCase) | `applyDiscount.test.ts` | | `describe()` | Ticket / User Story | `'{TICKET-ID}: Validate {feature}'` | `'UPEX-411: Validate discount codes'` | | `test()` | Scenario / test case | `'{TICKET-ID}: should {behavior} when {condition}'` | `'UPEX-411: should apply percentage discount when code is valid'` | Every `test()` must include the ticket ID as a prefix. `describe` blocks may include the ticket ID when the file is tied to a single ticket. --- ## 2. ATC Design Philosophy This section codifies the rules that determine what is an ATC, when two scenarios collapse into one ATC, and how assertions relate to test cases. These rules are the source of truth for KATA test design. ### What is an ATC (and what is NOT) An ATC represents a **complete action** that changes or validates system state. It maps 1:1 with a test case ticket in the TMS via `@atc('TICKET-ID')`. | Type | What it does | Example | Has `@atc`? | |------|-------------|---------|-------------| | **ATC** | Action that changes system state | `authenticateSuccessfully()`, `createOrderSuccessfully()` | Yes | | **Helper** | Reads data (no state change) | `getOrders(filters)`, `getCurrentUser()` | No (use `@step`) | A simple GET is a helper, not an ATC. A GET that verifies an action's outcome belongs **inside** the ATC as a verification step: ```typescript @atc('TICKET-ID') async createOrderSuccessfully(orderData) { // ACTION const [response, body, sent] = await this.apiPOST(...); // VERIFICATION (GET to confirm persistence) const [, persisted] = await this.apiGET(`/orders/${body.id}`); // ASSERTIONS expect(response.status()).toBe(201); expect(persisted.id).toBe(body.id); return [response, body, sent]; } ``` ### TC Identity Rule: Precondition + Action A test case's identity is determined by exactly two elements: 1. **Precondition**: the state the system must be in 2. **Action**: the user trigger **All expected results** from the same (precondition, action) pair belong to the **same TC** — regardless of which panel, endpoint, or UI section they validate. ``` TC Identity = Precondition + Action | All expected outputs are assertions of THIS TC ``` | Different TC? | Reason | |--------------|--------| | Yes | Different **precondition**: product out-of-stock vs in-stock | | Yes | Different **action**: open detail page vs click "Add to Cart" | | Yes | Different **equivalent partition**: percentage discount vs fixed-amount | | **No** | Same precondition + action, checking pricing block vs reviews block | | **No** | Same precondition + action, checking one more field in the response | **Anti-pattern — splitting by concern:** ``` // WRONG: 3 separate TCs for the same input TC-A: Open published product -> verify Pricing block values TC-B: Open published product -> verify Reviews block values TC-C: Open published product -> verify detail page structure // These share the SAME precondition and action -> they are ONE TC ``` **Correct — one TC, all assertions:** ``` TC: Open product detail page for published in-stock product Precondition: Product is published and has stock > 0 Action: Navigate to product detail page Expected Output: - Page structure visible (heading, image gallery, reviews section) - Pricing values correct (base - discount = final) - Inventory metrics correct (stock count, delivery estimate) - Add to Cart button enabled - Reviews block shows rating + review count ``` ### Equivalence Partitioning Inputs that produce the **same output type** collapse into one parameterized ATC. Inputs that produce **different outputs** require separate ATCs. | Same ATC (parameterize) | Different ATC (create new) | |--------------------------|---------------------------| | Different **data** but same **behavior** | Different **actions** or **behavior** | | All inputs produce the same output type | Outputs are fundamentally different | | Buy 1 product vs buy 5 products (same checkout flow) | Credit card checkout vs bank transfer (different steps) | | Minor output variation -> use conditionals sparingly | Different endpoint, UI flow, or assertion set | ```typescript // WRONG: three ATCs that all produce HTTP 401 @atc('T1') async loginWithWrongEmail() { /* -> 401 */ } @atc('T2') async loginWithWrongPassword() { /* -> 401 */ } @atc('T3') async loginWithEmptyFields() { /* -> 401 */ } // RIGHT: one parameterized ATC @atc('T1') async loginWithInvalidCredentials(payload: LoginPayload) { // All variations produce 401 with an error message const [response, body] = await this.apiPOST<ErrorResponse, LoginPayload>('/auth/login', payload); expect(response.status()).toBe(401); expect(body.error).toBeDefined(); return [response, body, payload]; } ``` **Rule of thumb**: if the **actions** inside the ATC change, it is a different ATC. If only the **data** changes but the system behaves identically, it is the same ATC. > **EP-merge collapses WITHIN a partition — never across.** Parameterizing the three invalid-credential inputs above into one 401 ATC is correct: they share a partition. It is a *defect* to use the same merge to swallow distinct partitions, boundaries, or states. A valid login (→ 200), a locked account (→ 423), and a value at `max+1` (→ 400 boundary) are **separate ATCs** — merging them loses coverage. EP is a 1:N expansion tool first (one ATC per partition) and a deduplication tool second (one ATC within a partition). See `agentic-qa-core/references/test-design-doctrine.md`. > **EP does not replace BVA.** Same-behavior merge hides off-by-one defects. Wherever a field has a range / limit / length / date-window, add explicit boundary ATCs (`min-1·min·min+1 … max-1·max·max+1`, plus zero / empty / null) — these are *distinct partitions at the edges*, so they are separate (often parameterized) ATCs, not folded into the happy-path case. ### Tests validate FLOWS, not individual properties Do not create N tests checking N fields of the same response. One test validates the complete flow with multiple assertions. ```typescript // WRONG: 3 tests for the same API call test('should return orders', async ({ api }) => { const orders = await api.orders.getOrders(filters); expect(orders.length).toBeGreaterThan(0); }); test('should have referenceNumber', async ({ api }) => { const orders = await api.orders.getOrders(filters); // same call expect(orders[0].referenceNumber).toBeDefined(); }); // RIGHT: one test, multiple assertions on the same flow test('TICKET-ID: should create order with correct totals when discount applied', async ({ api }) => { const [, order] = await api.orders.createOrderSuccessfully(orderData); const totals = await api.orders.getTotals({ orderId: order.id }); expect(order.id).toBeDefined(); expect(order.discountApplied).toBe(true); expect(totals.finalAmount).toBe(totals.baseAmount - totals.discountAmount); }); ``` Separate tests only when the **scenario is fundamentally different** (different flow, different preconditions, different user role) — not when checking a different field of the same response. ### Assertion layers Assertions exist at two levels and serve different purposes: ``` Test Flow | +-- ATC 1: createOrderSuccessfully() | +-- [ATC assertions: status 201, order persisted] | +-- ATC 2: applyDiscountSuccessfully() | +-- [ATC assertions: discount applied, total recalculated] | +-- Test-level assertions: +-- [Final state: total matches base - discount, tax correct] ``` - **ATC assertions** (fixed, inside the ATC): validate that the individual action succeeded. These run every time the ATC is called, in every test. - **Test-level assertions** (in the test file): validate the overall outcome after combining multiple ATCs. These are specific to the scenario. Assertions are checkpoints along a flow, not the purpose of the test. The test is the journey; assertions are road signs. --- ## 3. Folder Structure ``` /config variables.ts # Single source of truth for env + URLs /tests /components TestContext.ts ApiFixture.ts UiFixture.ts TestFixture.ts /api ApiBase.ts {Resource}Api.ts /ui UiBase.ts {Page}Page.ts /steps {Domain}Steps.ts /data /fixtures # JSON/CSV for parameterization (commit) /mocks # Mock/stub API responses {endpoint}/{METHOD}.{status}.json (commit; create-on-demand) /uploads # Files for upload tests (commit) /downloads # Download destination (gitignore) /integration /{module}/{verbFeature}.test.ts /e2e /{module}/{verbFeature}.test.ts /setup global.setup.ts /teardown global.teardown.ts /utils decorators.ts KataReporter.ts /test-results # Playwright artifacts (gitignore) /screenshots /videos /traces /playwright.config.ts ``` ### Playwright artifact configuration ```typescript // playwright.config.ts export default defineConfig({ outputDir: 'test-results', retries: 0, use: { screenshot: 'only-on-failure', video: 'retain-on-failure', trace: 'retain-on-failure', }, }); ``` `test-results/` and `tests/data/downloads/` are gitignored. `tests/data/fixtures/` and `tests/data/uploads/` are committed. `tests/data/mocks/` is committed when it exists, but is create-on-demand — it is not present until a scope needs canned mock responses. ### Test module folders (`{module}`) The `{module}` segment under `tests/integration/` and `tests/e2e/` is the **business domain**, named in **kebab-case, plural where natural**. It maps 1:1 to the Jira Component / product area, so the folder name and the Component label stay aligned. | Good | Why | |------|-----| | `orders/`, `users/`, `payments/` | Business domain, plural where natural | | `user-management/`, `checkout-flows/` | Multi-word domain, kebab-case | Do not name a module folder after a UI section, a page, or a sprint (`tab-2/`, `Q3/`, `LoginScreen/`). The same `{module}` value is reused by the spec scaffolding (`{PREFIX}` abbreviation in `planning-playbook.md`) and by the Jira Component, so it must read as a product area. --- ## 4. Tagging Strategy Use Playwright tags on `test()` and `describe()` to group runs and drive the CI matrix. | Tag | Meaning | Typical use | |-----|---------|-------------| | `@critical` | Core user journey (login, checkout, payment). Blocks release if fails. | Smoke suite, gated deploys. | | `@smoke` | Post-deploy health check. Runs on every deployment. | Smoke workflow. | | `@regression` | Full coverage. Runs nightly / pre-release. | Regression workflow. | | `@e2e` | End-to-end (UI + API). | Scope selection in CI. | | `@integration` | API / integration tests. | Scope selection in CI. | | `@flaky` | Known intermittent — under stabilization. | Excluded from `@critical` runs. | Example: ```typescript test.describe('TICKET-ID: Validate discount codes @regression', () => { test('TICKET-ID: should apply percentage discount when code is valid @critical', async ({ api }) => { ... }); }); ``` Run selection: ```bash bun run test --grep "@critical" bun run test --grep "@smoke" bun run test --grep "@regression" bun run test --grep-invert "@flaky" ``` Every `@critical` test is also `@smoke` in practice. Tags are additive — you can carry multiple tags on the same test. --- ## 5. Component Structure ### File template (API component) ```typescript /** * KATA Layer 3 — {Resource} API Component */ import { expect, type APIResponse } from '@playwright/test'; import { ApiBase } from '@api/ApiBase'; import { atc, step } from '@utils/decorators'; import type { TestContextOptions } from '@TestContext'; // ============================================ // Types // ============================================ export interface ResourcePayload { ... } export interface ResourceResponse { ... } // ============================================ // Component Class // ============================================ export class ResourceApi extends ApiBase { constructor(options: TestContextOptions) { super(options); } // ─── HELPERS (read-only, @step for tracing) ──────────── @step async getResourceById(id: string): Promise<[APIResponse, ResourceResponse]> { return this.apiGET<ResourceResponse>(`/api/resource/${id}`); } // ─── ATCs (state-changing, @atc for TMS) ─────────────── @atc('TICKET-ID') async createResourceSuccessfully(payload: ResourcePayload): Promise<[APIResponse, ResourceResponse, ResourcePayload]> { const [response, body, sent] = await this.apiPOST<ResourceResponse, ResourcePayload>( '/api/resource', payload); expect(response.status()).toBe(201); expect(body.id).toBeDefined(); return [response, body, sent]; } } export default ResourceApi; ``` ### Method order inside a component 1. Constructor 2. Shared locators (UI only, `private readonly`) 3. Navigation methods (UI only) 4. Helpers (no decorator, optional `@step`) 5. ATCs (`@atc('TICKET-ID')`) ### AAA (Arrange-Act-Assert) inside an ATC ```typescript @atc('TICKET-ID') async signInSuccessfully(payload: SignInPayload): Promise<[APIResponse, AuthResponse, SignInPayload]> { // ACT — perform the action const [response, body, sent] = await this.apiPOST<AuthResponse, SignInPayload>( '/auth/signin', payload); // ASSERT — fixed assertions validate the action succeeded expect(response.status()).toBe(200); expect(body.user).toBeDefined(); expect(body.session.access_token).toBeDefined(); // RETURN — for chaining with other ATCs return [response, body, sent]; } ``` "Arrange" for ATCs is usually empty — preconditions are passed in as parameters, not built inside the ATC. ### Return types ```typescript // API GET Promise<[APIResponse, TBody]> // API POST/PUT/PATCH Promise<[APIResponse, TBody, TPayload]> // UI ATC Promise<void> ``` API ATCs return tuples to enable chaining; UI ATCs typically return `void` because assertions are inside the ATC. When a UI ATC must expose data to the test file (e.g., a generated order ID from the confirmation page), return a typed object. ### Docstrings Component class: ```typescript /** * KATA Layer 3 — Auth API Component. * Handles authentication operations: sign in, sign out, user profile. */ ``` ATC methods: ```typescript /** * Sign in with valid credentials. * Returns: [APIResponse, AuthResponse, SignInPayload] */ @atc('TICKET-ID') async signInSuccessfully(payload: SignInPayload) { ... } ``` --- ## 6. Test Data Strategy KATA distinguishes three sources of test data. Classify every data need in the plan before coding. ### Pre-Execution variables (static) Defined in `config/variables.ts` and `.env` before the run: Credentials are env-prefixed per environment and selected by `TEST_ENV` (the real pattern in `config/variables.ts`): ```typescript // config/variables.ts (real pattern, simplified) const { TEST_ENV = 'local', LOCAL_USER_EMAIL, LOCAL_USER_PASSWORD, STAGING_USER_EMAIL, STAGING_USER_PASSWORD } = process.env; const userCredentialsMap: Record<Environment, { email: string, password: string }> = { local: { email: LOCAL_USER_EMAIL ?? '', password: LOCAL_USER_PASSWORD ?? '' }, staging: { email: STAGING_USER_EMAIL ?? '', password: STAGING_USER_PASSWORD ?? '' }, }; export const config = { testUser: userCredentialsMap[TEST_ENV as Environment], // ... }; ``` Tests read `config.testUser` from `@variables` — never `process.env` directly. Rule: credentials come from `.env` only (only the current `TEST_ENV`'s pair is required). Never hardcode, never check into git. ### Dynamic variables (runtime) Generated during test execution using `TestContext` utilities (typically `faker`): ```typescript test('TICKET-ID: should create user with dynamic data', async ({ api }) => { const userData = api.generateUserData(); await api.users.createUserSuccessfully(userData); }); ``` Rule: every test generates its own unique data so parallel runs and retries never collide. ### Fixture files (parameterisation) For data-driven tests, load from JSON/CSV in `tests/data/fixtures/`. **File naming**: `{resource}-{variant}.json` — lowercase, kebab-case. The `{resource}` is the entity (`users`, `orders`, `payments`); the `{variant}` names the scenario or partition the rows belong to, so reuse is explicit at the import site. | File | Holds | |------|-------| | `users-valid.json` | Valid user rows for happy-path parameterization | | `orders-boundary.json` | Boundary-value rows (BVA) for order limits | | `payments-error-cases.json` | Invalid payment inputs expecting error responses | ```typescript import usersData from '@data/fixtures/users-valid.json'; for (const user of usersData) { test(`TICKET-ID: should signup ${user.type} user`, async ({ ui }) => { await ui.signup.signupWithValidCredentials({ email: user.email, password: user.password, }); }); } ``` Data flow: files → test arguments → ATC arguments. ### Mock / stub response files When a test intercepts a backend call and supplies a canned response (Playwright `route.fulfill()`), the response body lives as a static fixture under `tests/data/mocks/`, named `tests/data/mocks/{endpoint-path}/{METHOD}.{status}.json`. The endpoint path mirrors the API route (discoverable); the HTTP method and status code in the filename signal exactly which interaction and outcome the mock stands in for. | File | Stubs | |------|-------| | `tests/data/mocks/auth/login/POST.200.json` | Successful login response | | `tests/data/mocks/users/POST.201.json` | User-creation success | | `tests/data/mocks/users/create/POST.400.json` | User-creation validation error | ```typescript import loginOk from '@data/mocks/auth/login/POST.200.json'; await page.route('**/auth/login', route => route.fulfill({ status: 200, json: loginOk })); ``` Use mocks only to isolate the unit under test from an unavailable or non-deterministic dependency — prefer real backend calls (the Discover → Modify → Generate strategy above) for functional coverage. ### Discover → Modify → Generate The full data strategy (when to reuse staging data, when to mutate it, when to generate fresh) lives in `test-data-management.md`. At the standards level, three rules hold: 1. **Never assume data exists.** If a test needs a user, either create one via API or generate from faker. 2. **Never mutate someone else's data.** If a test updates a record, it must create that record first. 3. **Never share state between tests.** Parallel runs depend on it. --- ## 7. Stability (Anti-Flakiness) ### No retries by default ```typescript // playwright.config.ts export default defineConfig({ retries: 0 }); ``` Retries mask real issues. Passing on retry is a red flag, not a success. Fix the root cause. ### No hardcoded waits ```typescript // WRONG await page.waitForTimeout(3000); // RIGHT await page.waitForSelector('[data-loaded="true"]'); await page.waitForLoadState('networkidle'); await page.waitForResponse(resp => resp.url().includes('/api/data') && resp.ok()); ``` ### Conditional waits for unpredictable UI ```typescript const popup = page.locator('[role="dialog"]'); const isVisible = await popup.isVisible({ timeout: 2000 }).catch(() => false); if (isVisible) await popup.locator('button:has-text("Close")').click(); ``` ### Intercept the real backend call ```typescript await Promise.all([ page.waitForResponse(resp => resp.url().includes('/api/cart')), page.locator('[data-testid="add-to-cart"]').click(), ]); ``` ### Soft fail (use sparingly) ```typescript @atc('TICKET-ID', { softFail: true, severity: 'normal' }) // Allure's vocabulary: blocker | critical | normal | minor | trivial. `'medium'` is a type error. async verifyOptionalField() { ... } ``` | Use soft fail | Do not use soft fail | |---------------|----------------------| | Optional form fields | Critical functionality | | Exploratory tests | Blocking validation | | Non-critical features that should not stop the flow | When failure means subsequent ATCs are meaningless | ### Blocked by a known bug When a test cannot pass because of a **known product bug** (not a test bug, not flakiness), mark it `test.fail('Blocked by {BUG-KEY}')` AND tag it `@blocked:{BUG-KEY}`: ```typescript test('TICKET-ID: should reject signup with invalid email @blocked:UPEX-999', async ({ api }) => { test.fail(true, 'Blocked by UPEX-999'); // server-side validation missing await api.users.signupWithInvalidEmail(payload); }); ``` The test stays in the suite (no silent skip) so the failure remains informative, and the `@blocked:{BUG-KEY}` tag lets regression runs filter it out of GO/NO-GO until the bug is fixed. Remove both markers when the bug closes and the test goes green. | Marker | Means | Use when | |--------|-------|----------| | `test.fail('Blocked by {BUG-KEY}')` + `@blocked:{BUG-KEY}` | A known product bug prevents passing | The product is broken — a bug key exists | | `softFail` (decorator option) | Non-critical assertion tolerated | Optional field / exploratory check that should not stop the flow | | `@flaky` (tag) | Intermittent, under stabilization | The test itself is unstable — no product bug | These three are distinct: `@blocked:{BUG-KEY}` blames the product (a filed bug), `softFail` tolerates a non-critical assertion, and `@flaky` flags an unstable test. Never use one in place of another. (Regression GO/NO-GO consumption of `@blocked:{BUG-KEY}` is documented in `regression-testing`.) --- ## 8. Error Handling ### Inside ATCs (the methods a test can reach) Fail fast with a descriptive error. ```typescript async apiGET<T>(endpoint: string): Promise<[APIResponse, T]> { if (!this.request) { throw new Error('Request context not set. Ensure fixture provides request.'); } ... } ``` ### Inside utilities (private helpers) Silent fail — return `null` / `undefined` for missing data. ```typescript private async parseResponseBody<T>(response: Response): Promise<T | null> { try { return (await response.json()) as T; } catch { return null; } } ``` Never swallow errors inside an ATC without re-throwing — the test must fail visibly. --- ## 9. Code Quality ### Linting ESLint with `@antfu/eslint-config` (flat config): ```bash bun run lint:check # Check for issues bun run lint:fix # Auto-fix issues ``` ### Type Checking TypeScript with relaxed mode (no `experimentalDecorators`): ```bash bun run types:check # tsc --noEmit ``` ### Import aliases (mandatory) ```typescript // RIGHT import { config, env } from '@variables'; import { ApiBase } from '@api/ApiBase'; // WRONG import { config } from '../../../config/variables'; ``` See `typescript-patterns.md` for the full alias list. --- ## 10. Review Checklists Every component and every test file is gated by a checklist before merge. Paste this into PR reviews. ### Component review - [ ] File name is PascalCase and matches the class name. - [ ] Class extends `ApiBase` or `UiBase` (never `TestContext` directly in Layer 3). - [ ] Constructor accepts `TestContextOptions` and passes to `super()`. - [ ] Helpers at top (no decorator or `@step`), ATCs below with `@atc('TICKET-ID')`. - [ ] ATCs have tuple return type (API) or `Promise<void>` (UI). - [ ] Every ATC contains fixed assertions. - [ ] Imports use aliases only, no relative paths. ### ATC review - [ ] Name follows `{verb}{Resource}{Scenario}` camelCase. - [ ] Represents a complete test case (mini-flow), not a single `page.click()`. - [ ] Different expected outputs = different ATCs; same output with different data = one parameterized ATC (equivalence partitioning). - [ ] Locators inline unless used in 2+ ATCs of this component (then `private readonly`). - [ ] No ATC calls another ATC. Chains live in `tests/components/steps/`. - [ ] Max 2 positional parameters; 3+ use an object parameter. - [ ] Return type annotated explicitly. - [ ] `@atc('TICKET-ID')` references a real TMS ticket. - [ ] No hardcoded waits (`waitForTimeout`). - [ ] Sensitive parameters (password, token) named correctly so decorators mask them. ### Test file review - [ ] File under `tests/integration/{module}/` or `tests/e2e/{module}/`. - [ ] Name follows `{verb}{Feature}.test.ts` camelCase with a user-action verb. - [ ] Every `test()` has the ticket ID prefix and `should {behavior} when {condition}` format. - [ ] Uses the correct fixture: `{ api }`, `{ ui }`, or `{ test }` (Steps classes are instantiated directly, not requested as a fixture). - [ ] Creates its own test data — no shared state with other tests. - [ ] Test-level assertions validate business logic, not individual fields of a single response. - [ ] Tags applied where needed (`@critical`, `@smoke`, `@regression`). - [ ] No relative imports. - [ ] Component used is registered in the relevant fixture. --- ## 11. Anti-Patterns Common mistakes that fail code review. | Anti-pattern | Why it is wrong | Fix | |--------------|-----------------|-----| | ATC that only wraps one `page.click()` | Not a test case, just an interaction | Merge into a complete ATC flow | | Separate `locators/*.ts` file | Maintenance overhead, disconnected from tests | Inline in ATC, or `private readonly` on class for 2+ uses | | Multiple ATCs with the same expected output | Violates equivalence partitioning | One parameterized ATC | | Helper that wraps a single `page.fill()` | Playwright already does it | Delete helper, call `page.fill()` inline | | ATC calling another ATC | Breaks atomicity and traceability | Use Steps module | | `waitForTimeout(3000)` | Arbitrary, flaky, slow | Wait for specific condition | | Relying on retries (`retries > 0`) | Masks real issues | Investigate failure, fix root cause | | Shared state between tests | Tests become order-dependent | Each test creates its own data | | Component not registered in fixture | Tests cannot access it | Add to `ApiFixture` / `UiFixture` (Steps classes need no registration) | | Relative imports (`../../../config`) | Breaks lint, hurts refactors | Use alias (`@variables`, `@api/`, ...) | | Test name without ticket ID | Breaks TMS traceability | `test('TICKET-ID: should ... when ...')` | | Six tests checking six fields of one response | Violates TC Identity rule | One test with multiple assertions | ### Worked anti-pattern examples ```typescript // ATC with a single interaction — WRONG @atc('TICKET-ID') async clickAddToCartButton() { await this.page.click('[data-testid="add-to-cart"]'); } // Multiple ATCs, same output (HTTP 401) — WRONG @atc('T1') async loginWithWrongEmail() { /* -> 401 */ } @atc('T2') async loginWithWrongPassword() { /* -> 401 */ } @atc('T3') async loginWithEmptyFields() { /* -> 401 */ } // Separate locator file — WRONG // locators/checkout.ts export const LOCATORS = { addToCartBtn: '[data-testid="add-to-cart"]', cartTotal: '[data-testid="cart-total"]', }; // Helper wrapping a one-liner — WRONG private async fillEmail(email: string) { await this.page.locator('#email').fill(email); } ``` --- ## 12. Quality Gates (Must Pass Before Merge) These gates block PR merge. Run them locally before opening the PR. | Gate | Command | Must be | |------|---------|---------| | Tests pass | `bun run test <path>` | All green, zero retries used | | Type check | `bun run types:check` | No errors | | Lint | `bun run lint:check` | No errors | | Component registered | visual | In relevant fixture | | ATC IDs linked | visual | Every `@atc('X')` maps to a real TMS test case | | Tags correct | visual | `@critical` / `@smoke` / `@regression` applied as planned | | No AI attribution in commits | `git log` | No "Co-Authored-By: Claude" or similar | ### Validation loop ```bash # 1. Write code # 2. Run the new test bun run test tests/integration/orders/createOrder.test.ts # 3. If green, check types bun run types:check # 4. If clean, lint bun run lint:check # 5. If clean, extract manifest and verify ATC IDs bun run kata:manifest # 6. Open PR, confirm reviewer checklist passes ``` Order matters. Do not chase lint errors before tests pass — the test may remove the problematic code. --- ## 13. Complementary Testing (Optional) KATA covers functional testing. For other testing types, integrate as needed without breaking the architecture. | Type | Tools | When to use | |------|-------|-------------| | Visual regression | Playwright `toHaveScreenshot`, Percy, Chromatic | Design-heavy applications, component libraries | | Accessibility | `@axe-core/playwright` | Public-facing apps, compliance requirements | | Performance | Lighthouse CI, k6, Playwright Performance API | SLAs, performance-critical paths | These live alongside functional tests but are out of KATA's core scope. Add them when the project requires them; do not force them into ATCs. --- ## 14. Quick Reference **Validation loop** — run for every task, in this order: ```bash # 0. PRE-CODE — load kata-manifest.json (Critical Rule #12). # Confirm proposed Components and ATC IDs are not duplicates. cat kata-manifest.json | jq '.components, .summary' # or open in editor # 1-3. Code → tests → types → lint bun run test <path> bun run types:check bun run lint:check # 4. If components or ATCs were added/changed, regenerate the manifest. bun run kata:manifest # 5. POST-CODE — stage the manifest if step 4 changed it, then validate # that the husky pre-commit gate would pass. git add kata-manifest.json bun run kata:manifest:check # exits 1 if the committed manifest is stale ``` ```bash # OpenAPI schema sync (if API component types come from spec) bun run api:sync # Environment setup cp .env.example .env # populate test credentials ``` File path shapes: ``` tests/components/api/{Resource}Api.ts tests/components/ui/{Page}Page.ts tests/components/steps/{Domain}Steps.ts tests/integration/{module}/{verbFeature}.test.ts tests/e2e/{module}/{verbFeature}.test.ts ``` -
batch-fleet.md 14.7 KB
# Batch Fleet — one automation batch, N parallel workers The default executor for a batch is **this session, one Work Package at a time**. This file covers the second executor: a **conductor** session that partitions the batch and hands each part to a **worker** — a persistent session of its own, in its own worktree, running `/test-automation` on its slice. It is a scoping and integration contract. The transport — how a worker is launched, supervised, asked a question, and closed — belongs to `orca-orchestration/SKILL.md` and is written here as `[ORCHESTRATION_TOOL] <verb>: …` pseudocode. Load that skill before firing any of it. Nothing in this file is required to automate a batch. A fleet is faster when the batch really is parallel and strictly worse when it is not: the failure mode is not a slow suite, it is two sessions rewriting the same fixture file. --- ## 0 · Is a fleet the right executor? | Batch shape | Executor | |---|---| | 1 Work Package | this session. No conductor, no worker, no launch file | | 2 WPs, same module | this session, sequentially. They share components — parallelism buys nothing and costs a merge | | 3+ WPs over **disjoint** modules | fleet is available. Offer it; never enter it silently | | 3+ WPs, all in one module | this session, sequentially. Partitioning is impossible (see §2) | | User asked for parallel sessions | fleet, whatever the shape — but say which WPs will serialize inside a worker and why | A fleet also needs a conductor that will stay alive for the whole batch. If the session that partitions the work is going to be closed, do not launch workers: write the launch file, hand it over, and let the next session adopt the run. --- ## 1 · The unit of delivery: a Work Package A **Work Package (WP)** is one delivery unit = **one `test-specs/<ID>/` spec**: the `spec.md`, `automation-plan.md`, and `atc/*.md` under `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<ID>/`. One WP = one Plan → Code → Review pass = one reviewable branch. | WP origin | What `<ID>` is | Where its TC list comes from | |---|---|---| | **Sprint** | the Story key + slug | the Story's `Candidate` verdicts from `/test-documentation` (Stage 4). `Manual` / `Deferred` never enter a WP | | **Discovery** | the Tech Story key + slug, or the module slug | a coverage gap recorded as a Tech Story; the WP's spec is authored in Phase 1 like any other | Rules: - **A batch is a list of WPs, never a list of files or a list of TCs.** A TC that is not inside a WP is not in the batch. - A WP that has no `spec.md` yet is **not ready to dispatch**. Planning is the conductor's or the worker's Phase 1 — but it happens before the WP counts as assignable work, because the partition in §2 is computed from the plan's component list. - One WP never spans two modules. If its plan touches two, split it into two WPs before partitioning; a cross-module WP pins two modules to one worker and serializes the rest of the batch behind it. --- ## 2 · Partition: component ownership by module **Partition by module, not by ticket.** A worker owns a **module's components** — every `Api`, `Page`, and `Steps` class of that module, plus its spec directories — for the whole batch. Every WP touching that module goes to that worker and they run **sequentially inside it**. ``` batch = [WP-1 … WP-n] for each WP: modules(WP) = modules of the components its automation-plan.md creates or edits group WPs by module -> one worker per module group | +- a worker's WPs run one after another, Plan -> Code -> Review each +- two workers never share a module +- a module group larger than the rest of the batch is the batch's critical path: say so up front ``` **Never two workers on the same module.** Same-module WPs share Pages, Apis, fixtures and locators, and they collide in exactly the files with the least merge tolerance: | Shared surface | Why it collides | Rule | |---|---|---| | `tests/components/api/*Api.ts`, `tests/components/ui/*Page.ts` | two workers add methods to the same class; both diffs are correct and the merge is a manual reconstruction | module ownership. One worker, one module | | `tests/components/steps/*Steps.ts` | a Steps chain is shared across modules by design | the conductor owns cross-module Steps edits, or one named worker does and the others declare a claim (§8) | | `tests/components/ApiFixture.ts`, `tests/components/UiFixture.ts` | every new component registers itself here, so **every** worker edits these two files | expect a conflict at integration and resolve it **additively** — keep both registrations, alphabetical, never "theirs" | | `kata-manifest.json` | generated; two workers regenerate different truths | never hand-merged. The conductor regenerates it per integration (§6) | | `api/schemas/` | generated by `bun run api:sync` | conductor-only (§4). A worker that needs a fresh schema asks for it | | `.env`, `.auth/` | one writer, many readers | conductor-only (§4) | --- ## 3 · Topology: one worktree per worker This skill writes code, so the topology is **one worktree per worker** — not several sessions in one checkout. Two sessions in one checkout contend on the single git index even when their files are disjoint, and the loser sees `index.lock` or a partially staged commit. - The conductor creates the worktree and **provisions it before the worker is launched**: a fresh worktree has the tracked files of its base and nothing else. `bun run worktree:provision` covers the gap (`.env`, dependencies, the harness skill alias, community skills, `.auth/`); the Jira cache is rebuilt separately with `bun run context:hydrate`. Full gap table: `orca-orchestration/references/provisioning.md`. - Verify the new worktree's HEAD against the remote base before handing it over. A worktree cut from a stale local ref is born behind, and the worker's first push is a surprise merge. - `.session/` is not in a worktree. Briefs, the roster, and the launch file live in the **primary checkout** and are cited to the worker by absolute path. - Creating, provisioning, launching, and **delivering the brief** are one indivisible operation. A launched worker with no brief idles, and idle looks identical to working. --- ## 4 · Conductor-only operations These are one-writer surfaces. A worker that runs them races every sibling: | Operation | Why conductor-only | |---|---| | `bun run api:login` (token mint) | the default token file is shared, and concurrent writers interleave. The conductor mints once and workers only read it; when a worker genuinely needs its own credential, the conductor mints an isolated set for it (`bun run api:login <env> --profile <label>`) — never a second concurrent login against the default path | | `bun run api:sync` (OpenAPI types) | regenerates `api/schemas/` for everyone | | issue-tracker / TMS **writes** — TC transitions, labels, `test_automation` links | rate limits, and two sessions transitioning the same TC produce a contradictory history | | `bun run kata:manifest` at integration time | see §6 | | worktree creation and removal | includes the orphan audit before removal (`.agents/skills/git-flow-master/references/worktrees.md`) | A worker that needs one of these **asks** — it does not do it. Workers still run their own local gate (`bun run test` / `types:check` / `lint:check`) inside their own worktree; that is not shared state. --- ## 5 · The per-worker brief The brief is the 7-component briefing from `agentic-qa-core/references/briefing-template.md` plus the fleet fields. Write it to a file in the primary checkout and give the worker its absolute path — a long brief pasted into a message gets truncated. Fleet-specific fields and the reporting protocol: `orca-orchestration/references/brief-template.md` and `orca-orchestration/references/worker-contract.md`. What a `/test-automation` worker's brief must nail down beyond the generic seven: 1. **Its WPs, in order**, each with the path of its `spec.md` / `automation-plan.md`. One WP at a time; no reordering, no scope growth. A WP that turns out to touch another worker's module → stop and report, never take it. 2. **File ownership**: the module's components it owns, and the explicit list of files it must NOT touch (`kata-manifest.json`, `api/schemas/`, another module's components). 3. **Fixture registration is expected to conflict** — register the component normally in `ApiFixture.ts` / `UiFixture.ts`, do not try to avoid the edit, do not reformat the file. 4. **Git**: branch name, base, whether it pushes and opens its own PR (§7). 5. **Scope-correction licence**: a plan that turns out wrong during Code is a legitimate finding. Record it in the WP's `spec.md`, report it, and wait — do not re-plan a different scope silently. 6. **Local gate before reporting**: `bun run test <paths>` → `bun run types:check` → `bun run lint:check`, in that order, green, plus the Phase 3 review checklist. A worker reports a red gate as a failure; it does not report "done, with notes". 7. **Reporting**: one completion message, an explicit success/failure outcome, the list of files it changed, and the path of its long report. Then stop — it does not close its own session and does not start the next WP unless the brief listed it. --- ## 6 · Integration The conductor integrates **one branch at a time**, in a fixed order it decides up front (biggest module group first — its critical path is the batch's), and treats each integration as a small review: 1. Confirm the worker's local gate was green and its branch is pushed. 2. Merge / PR the branch per §7. Resolve `ApiFixture.ts` / `UiFixture.ts` conflicts **additively**. 3. **Regenerate the manifest, never merge it**: `bun run kata:manifest`, then stage it so the freshness gate passes, then `bun run kata:manifest:check`. A conflicted `kata-manifest.json` is discarded and regenerated; a hand-resolved one is a lie about the tree. 4. Re-run the suite for the merged state (`bun run test`) plus `types:check` and `lint:check`. Two individually green branches can be red together — a duplicated `@atc` ID or a fixture name collision only exists after the merge. 5. Only then fire the TC lifecycle transitions for that WP (conductor-only, §4) and record the integration in the session ledger per `agentic-qa-core/references/session-management.md` §7. 6. Close the worker and remove its worktree **after** the orphan audit — gitignored files die with the worktree. A red merged state stops the integration queue. Do not keep merging branches onto a broken tree to "finish the batch". --- ## 7 · Git rules per strategy Resolve `git_strategy` from `.agents/project.yaml` first; `/git-flow-master` owns every command. | Strategy | Worker | Conductor | |---|---|---| | `sdet` | cuts `test/{KEY}-{slug}` **from the integration trunk**, runs the per-ticket loop, opens its PR into the trunk | creates and owns the trunk, merges each ticket PR `--no-ff` in integration order, runs the sync gate, opens the single final PR. Full runbook: `.agents/skills/git-flow-master/references/sdet-integration-trunk.md` | | `solo-main` | commits on its own `test/*` branch in its own worktree, pushes it | merges each branch (or PRs it) in integration order. Direct push to a protected branch still resolves `git_strategy.policy.direct_push_to_protected` | | anything else | own branch, own PR, base per strategy | merge order + the manifest regeneration of §6 | Invariants that no strategy relaxes: never rebase or force-push a branch a worker pushed; never `git add -A`; every commit carries the forensic trailers from `/git-flow-master` §3.2 and no AI attribution. --- ## 8 · Shared test data and fixtures Two workers in two worktrees still hit **one** environment and **one** database. Component ownership does not isolate a seeded account, a tenant, or a fixture row. Declare the intent at partition time (which WP reads which entity, which one writes it), and route the runtime arbitration through the claims protocol: `orca-orchestration/references/claims-protocol.md`. Two WPs that write the same entity either go to the same worker or serialize behind a claim. `test-data-management.md` still governs how each WP generates its own data — the cheapest fix for a contested fixture is a WP that generates its own. --- ## 9 · The launch file and the silence rule The conductor **always** writes the launch file — one self-contained line per worker — whether or not any orchestration binary exists on the machine. Supervised launch is the native path (`orca-orchestration`'s `worker-start --agent`); `launch.txt` is the payload for the human-paste fallback when nothing can launch it, and for the unsupervised custom-argv path in between. Identical payloads are what keeps the paths from drifting; a paraphrased line is the failure this rule exists to prevent. Shape, validation, and the three gated lines: `orca-orchestration/references/launch-seam.md`. The absence of an orchestration binary costs nothing and is never reported: not in the preflight gate, not in the plan, not in the review, and never named to the user from this skill. --- ## 10 · Gotchas | # | Gotcha | Symptom | Fix | |---|---|---|---| | 1 | Two workers on one module | `*Api.ts` / `*Page.ts` conflicts that cannot be merged mechanically | partition by module (§2); same-module WPs serialize in one worker | | 2 | `kata-manifest.json` merged by hand | the freshness gate passes on a manifest that describes no real tree; duplicate ATC IDs survive | regenerate per integration (§6) | | 3 | Fixture-registration conflict resolved with "theirs" | a component silently disappears from the fixture; tests fail with an undefined property | additive resolution only | | 4 | Worktree launched before provisioning | dependency errors, missing MCP servers, no credentials — all with the wrong error message | provision first (§3) | | 5 | Worker pushes nothing for 10 minutes | it never received the brief | check its worktree's `git status --porcelain`; re-deliver the brief | | 6 | Each worker mints its own token | interleaved writes to the shared token file; one worker 401s mid-suite | conductor mints once (§4) | | 7 | Worktree removed before the orphan audit | evidence, local reports, and `.env` gone with it | audit, copy out, then remove (`.agents/skills/git-flow-master/references/worktrees.md`) | | 8 | Both individually-green branches red together | duplicate `@atc` ID or colliding component name | re-run the suite on the merged state, per integration (§6) | | 9 | Browser sessions left open per worker | orphan Chromium processes eating GBs | every worker closes its browser sessions before reporting | --- ## 11 · Degraded path (no conductor available) A batch with no live conductor is still a batch. Run it in this session, WP by WP, in the order §2 would have produced (largest module group first). The partition is not wasted work: it is the order that minimizes rework, sequential or not. -
ci-integration.md 21.8 KB
# CI Integration — Playwright Config, Reporters, Projects Load when authoring or modifying `playwright.config.ts`, wiring a new reporter, tuning parallelism / sharding while authoring, or debugging why a test behaves differently locally versus in CI. This file is scoped to **Playwright-level** concerns — config, reporters, projects, sharding, environment propagation. Regression / smoke orchestration (GitHub Actions workflows, `gh run watch`, failure triage, release GO/NO-GO) is out of scope. That belongs to the `regression-testing` skill. If you are building the pipeline that runs these suites on a schedule or on PRs, start there. --- ## 1. `playwright.config.ts` — canonical shape The config has four load-bearing concerns: **projects**, **reporters**, **use** (global defaults), and **CI-vs-local switches**. Everything else is derived from these. ```typescript import { defineConfig, devices } from '@playwright/test'; import { config, env } from './config/variables'; // Single source of truth — the config NEVER reads process.env directly const baseURL = config.baseUrl; export default defineConfig({ testDir: './tests', testMatch: /.*\.test\.ts/, fullyParallel: false, forbidOnly: !!process.env.CI, // KATA doctrine: tests must be deterministic. A failure is investigated, // never masked by a retry — locally AND in CI. retries: 0, // Single worker for now — increase when tests are stable and parallelizable workers: 1, reporter: [ ['./tests/KataReporter.ts'], ['html', { outputFolder: 'playwright-report', open: 'never' }], ['json', { outputFile: 'test-results/results.json' }], ['junit', { outputFile: 'test-results/junit.xml' }], ['allure-playwright', { resultsDir: config.reporting.allureResultsDir /* + detail, categories, environmentInfo */ }], ], use: { baseURL, trace: 'retain-on-failure', // NOT `on-first-retry`: with `retries: 0` there is never a first retry, so a local failure would produce no trace at all screenshot: config.reporting.screenshotOnFailure ? 'only-on-failure' : 'off', video: env.isCI && config.reporting.videoOnFailure ? 'retain-on-failure' : 'off', headless: env.isCI || config.browser.headless, }, projects: [ { name: 'global-setup', testMatch: /global\.setup\.ts/, testDir: './tests/setup', teardown: 'global-teardown' }, // teardown wired as a PROJECT, not a globalTeardown hook { name: 'ui-setup', testMatch: /ui-auth\.setup\.ts/, testDir: './tests/setup', dependencies: ['global-setup'] }, { name: 'api-setup', testMatch: /api-auth\.setup\.ts/, testDir: './tests/setup', dependencies: ['global-setup'] }, { name: 'e2e', testMatch: '**/e2e/**/*.test.ts', dependencies: ['ui-setup'], use: { ...devices['Desktop Chrome'], storageState: config.auth.storageStatePath } }, { name: 'integration', testMatch: '**/integration/**/*.test.ts', dependencies: ['api-setup'], use: {} }, // ONE SMOKE PROJECT PER SURFACE. A single project spanning // `{e2e,integration}` has to pick ONE `use` block, and picking the UI one // hands a browser storageState — and therefore the session COOKIE — to API // tests. A test that clears the Bearer token to assert 401 then gets 200, // because the cookie still authenticates. Measured in this boilerplate: // the same test passed under `--project=integration` and failed under // `--project=smoke`. Each half mirrors its full-suite sibling plus the grep. { name: 'smoke-ui', grep: /@critical/, testMatch: '**/e2e/**/*.test.ts', dependencies: ['ui-setup'], use: { ...devices['Desktop Chrome'], storageState: config.auth.storageStatePath } }, { name: 'smoke-api', grep: /@critical/, testMatch: '**/integration/**/*.test.ts', dependencies: ['api-setup'], use: {} }, { name: 'global-teardown', testMatch: /global\.teardown\.ts/, testDir: './tests/teardown' }, ], outputDir: 'test-results', }); ``` Rules this template encodes: - **URLs come from `config/variables.ts`** — the config imports `config` / `env` and never reads `process.env` ad-hoc (single exception: the `CI` flag). See §5. - **`testMatch: /.*\.test\.ts/`** — only `*.test.ts` files run. A `*.spec.ts` file is silently ignored; name test files accordingly. - **Authentication via setup projects** — projects that need a logged-in state depend on `ui-setup` or `api-setup` and consume the generated `storageState` file. Do not login in `beforeEach` of every test. - **`forbidOnly: !!process.env.CI`** — `test.only` is legal locally (fast iteration) but fails the build on CI. Catches accidental commits. - **`retries: 0` everywhere** — locally and in CI. Tests are deterministic by doctrine; a failure is a signal to investigate, never something to mask with a retry. - **Serial execution is the shipped default** — `fullyParallel: false` + `workers: 1`. Parallelism is a deliberate future upgrade once the suite is proven stable, not a knob to flip casually. - **Teardown is a PROJECT** — `global-teardown` is activated by the `teardown:` property on `global-setup`, not by a `globalTeardown` hook and not by `dependencies`. --- ## 2. Projects dependency graph The setup projects gate the test projects. Visualised: ``` global-setup ├── ui-setup ──► e2e └── api-setup ──► integration (both → global-teardown) ``` ### 2.1 Generated auth artifacts ``` .auth/ ├── user.json storageState for E2E (cookies + localStorage) └── api-state.json JWT token + user metadata for integration tests ``` `.auth/` is gitignored. On a fresh checkout, the setup projects must run before either test project — which is exactly what the dependency chain encodes. If a contributor tries to run `bun run test:e2e` without authenticating, Playwright will invoke `ui-setup` automatically because of the `dependencies` array. ### 2.2 Project-scoped `use` Per-project `use` overrides global `use`. Common pattern: E2E sets `storageState`, integration does not. Desktop Chrome devices, viewports, and locale overrides also belong at the project level — never in individual tests. --- ## 3. Reporter chain — order matters The reporter array is load-bearing. Reporters run in order and the first one can mutate timing metrics for the rest. The canonical order is: 1. **`KataReporter`** (custom) — colored terminal tree, prints step boundaries, reads the NDJSON written by `@atc` to show per-ATC status. Goes first so its step events are registered before Allure grabs them. 2. **`html`** — Playwright's built-in HTML report. Lives in `playwright-report/`. `open: 'never'` prevents it from auto-opening locally (annoying in CI-like local runs). 3. **`json`** — machine-readable summary for tooling (`test-results/results.json`). 4. **`junit`** — XML for CI tools (`test-results/junit.xml`). Always on, local and CI. 5. **`allure-playwright`** — writes to `config.reporting.allureResultsDir`. The `bun run allure:generate` script post-processes this directory into a static site. 6. **`github`** (optional, commented out in the shipped config) — annotates PRs with failure locations when enabled in GitHub Actions. Not recommended with matrix strategies (errors multiply in the UI). ### 3.1 What `KataReporter` produces Terminal output only. It reads `step.category === 'test.step'` events from Playwright and prints a nested tree: ``` 🧪 Running Test [3/6] => UPEX-100: should be able to re-authenticate ---- ✓ ATC [PROJ-101]: authenticateSuccessfully({ email: "user@test.com", password: "***" }) ---- API OK: 200 - https://api.example.com/auth/login ✅ [PROJ-101] authenticateSuccessfully - PASS (825ms) ``` The final `atc_results.json` aggregation is written in `KataReporter.onEnd()`, not during individual tests — see the tracing reference for the pipeline. ### 3.2 Allure attachments `ApiBase` attaches every request/response pair to Allure automatically. `@atc` adds `allure.label`, `allure.severity`, `allure.link` based on the decorator argument. There is no manual `testInfo.attach()` or `allure.step()` needed in a well-written component — if you see them, flag in review. ### 3.2.1 Allure suite labels — single source of truth (Playwright tags) Allure suite / grouping labels are **derived from the Playwright tag** on the test — there is no separate Allure label taxonomy to maintain. The `@smoke` / `@regression` / `@e2e` / `@integration` / `@critical` tag is the **single source of truth**: it drives BOTH CI scope selection (`--grep @smoke`) AND Allure suite grouping. A test tagged `@integration` lands in the Allure `integration` suite/label automatically — do not add a duplicate Allure label for the same grouping. - **One tag, two consumers.** The same tag the CI workflow greps on is the tag Allure groups by. Never label a test `@integration` for grep and then hand-set a different Allure suite — they must agree by construction because they are the same string. - **No parallel taxonomy.** Do not invent Allure-only suite names. If a new grouping is needed, add it as a Playwright tag (so CI can select it) and let Allure inherit it. - **Where it is consumed downstream:** `regression-testing` reads the `suite` label off each Allure result during failure analysis — see `regression-testing/SKILL.md` (Phase 2 §Parse results). Because the label is tag-derived, the suite shown in a regression report is exactly the scope CI ran. ### 3.3 Artifact output paths | Artifact | Path | When | |----------|------|------| | HTML report | `playwright-report/` | Every run | | JSON summary | `test-results/results.json` | Every run | | JUnit XML | `test-results/junit.xml` | Every run | | Allure raw | `allure-results/` | Every run | | Traces | `test-results/{test}/trace.zip` | Per `trace` setting | | Screenshots on failure | `test-results/{test}/test-failed-*.png` | On failure | | Videos on failure | `test-results/{test}/video.webm` | Per `video` setting | | NDJSON partial | `reports/.atc_partial.ndjson` | During run, deleted in `onEnd()` | | Aggregated ATC results | `reports/atc_results.json` | After `KataReporter.onEnd()` | All paths except `reports/atc_results.json` should be gitignored. --- ## 4. Local vs CI configuration differences The deliberate divergences are few. **`retries` and `workers` are NOT among them** — `retries: 0` and `workers: 1` apply everywhere, local and CI. | Setting | Local | CI | Reason | |---------|-------|-----|--------| | `forbidOnly` | `false` | `true` | Let devs iterate with `.only`, block it on merge. | | `trace` | `'retain-on-failure'` | `'retain-on-failure'` | NOT a divergence. `'on-first-retry'` locally is a trap while `retries: 0`: there is no first retry, so the failure you just got produces no trace. A green run writes nothing either way. | | `video` | `'off'` | `'retain-on-failure'` (when `config.reporting.videoOnFailure`) | Same rationale. | | `headless` | `config.browser.headless` | forced `true` | CI runners have no display. | | `github` reporter | off | opt-in (commented in shipped config) | Writes PR annotations when enabled. | ### 4.1 The `retries: 0` rule — everywhere This is doctrine, not a local-only discipline. If a test passes on retry, it is flaky — the failure you would chase later is already in your diff. CI gets no retry allowance either: a test that needs a retry to pass in CI is a flakiness bug to fix, never something the config tolerates. Failure classification in `regression-testing` depends on this — a retry-masked pass would distort the FLAKY bucket of the GO/NO-GO analysis. ### 4.2 CI detection `!!process.env.CI` is the single switch. GitHub Actions, GitLab CI, CircleCI, Jenkins, and Buildkite all set `CI=true`. Do not check `process.env.GITHUB_ACTIONS` or other provider-specific variables inside the Playwright config — the config must stay provider-agnostic. --- ## 5. Environment variables the config reads The config reads through `config/variables.ts`, the single source of truth. It does not read `process.env` ad-hoc. | Env var | Consumed by | Purpose | |---------|-------------|---------| | `CI` | Playwright config | Toggle local vs CI switches | | `TEST_ENV` | `config/variables.ts` | Selects the environment entry in the internal URL map AND the credential set. `config.baseUrl` (frontend, feeds `use.baseURL`) and `config.apiUrl` (API host, feeds `ApiBase`) both come from that map — there are NO `BASE_URL` / `API_BASE_URL` env vars. | | `LOCAL_USER_EMAIL` / `LOCAL_USER_PASSWORD` | auth setup projects | Local credentials | | `STAGING_USER_EMAIL` / `STAGING_USER_PASSWORD` | auth setup projects | Staging credentials | | `AUTO_SYNC` | `jiraSync.ts`, the workflows' `Sync Results to TMS` step | Enable the TMS write-back. It runs as a step AFTER the test step, not inside the teardown (see atc-tracing reference) | | `TMS_PROVIDER` | `jiraSync.ts` | `xray` / `jira` / `none` | | `STP_EXECUTION_KEY` | `jiraSync.ts` | **Xray only.** Target of the write-back: the key of the **STR** Test Execution linked to the sprint STP — never the STP itself (the sync reads the issue type and refuses a Test Plan). Unset → each run mints a new, unparented Execution. | ### 5.1 Rules - **Never hardcode credentials** anywhere in the repo — `.env` only. Rejected at review. - **Never read `process.env` inside an ATC** — route through `@variables`. Keeps component code portable. - **`.env.example` is the contract** — every new variable must be added there with a placeholder; missing rows break onboarding. --- ## 6. Sharding and parallelism — while authoring The project-level sharding used in CI pipelines is out of scope here. This section covers what you do **locally** while authoring and debugging. ### 6.1 Parallelism tuning - **Default**: `fullyParallel: false` + `workers: 1` — serial execution is the shipped default, local and CI. Terminal output stays readable and shared-state bugs cannot hide behind interleaving. - **Stress-testing for future parallelism**: before proposing a workers bump, prove the suite survives concurrency by overriding at the command line. ```bash bun run test -- --workers=8 --repeat-each=5 ``` A test that passes serially but fails here has shared state — fix it before any config-level parallelism change. ### 6.2 Sharding while authoring Use sharding locally only when debugging CI-style execution order. Two-shard split: ```bash bunx playwright test --shard=1/2 bunx playwright test --shard=2/2 ``` Each shard gets a deterministic subset of the test files — useful for reproducing a "test fails only in shard 2" issue. The global `--shard` flag is orthogonal to `--project`: you can shard a single project with `--project=integration --shard=1/2`. ### 6.3 Merging reports from local shards ```bash bunx playwright merge-reports --reporter=html ./blob-report ``` Required only if you set `reporter: 'blob'` per shard. Normal authoring loops do not need this. --- ## 7. Command cheatsheet ### 7.1 Running tests ```bash # Run everything (all projects) bun run test # Run a single project bun run test -- --project=integration bun run test -- --project=e2e # Run a single file bun run test tests/integration/orders/createOrder.test.ts # Run by tag bun run test -- --grep @smoke bun run test -- --grep "@critical|@regression" bun run test -- --grep-invert @flaky # Run a single test title bun run test -- --grep "PROJ-101: should login" # UI mode (interactive watcher) bun run test:ui # Debug mode (opens inspector) bun run test -- --debug tests/e2e/login/login.test.ts # Headed mode bun run test -- --headed # Specific browser at runtime (overrides project device) bun run test -- --project=e2e --browser=firefox ``` ### 7.2 Reports ```bash # Open the HTML report from the last run bunx playwright show-report # Generate Allure site from allure-results/ bun run allure:generate # Produce kata-manifest.json (static registry of components / ATCs) bun run kata:manifest # Sync OpenAPI + regenerate types bun run api:sync ``` ### 7.3 Quality loop (local, before PR) ```bash bun run test <path> # 1. does the new test pass? bun run types:check # 2. no TS errors bun run lint:check # 3. no lint errors bun run kata:manifest # 4. registry updated, ATCs visible git add kata-manifest.json # 5. stage the manifest bun run kata:manifest:check # 6. confirm the husky pre-commit gate would pass ``` Run in order. Do not chase lint errors before tests pass — the failing test may delete the offending code. ### 7.4 Local quality loop — kata-manifest Two-command discipline for the test author: | Command | When | Effect | |---|---|---| | `bun run kata:manifest` | After adding/renaming a Component, ATC, or Steps method | Regenerates `kata-manifest.json` in place | | `bun run kata:manifest:check` | Before committing | Fails fast (exit 1) if the committed manifest is out of date | `.husky/pre-commit` runs `:check` automatically when staged files touch `tests/components/`, `scripts/kata-manifest.ts`, or `kata-manifest.json` itself. Commits that don't touch those paths skip the gate (no perf penalty). ### 7.5 Manifest troubleshooting | Symptom | Cause | Fix | |---|---|---| | `--check` exits 1 with "stale" | Component or ATC change not regenerated | `bun run kata:manifest && git add kata-manifest.json` | | `--check` exits 1 with "missing" | `kata-manifest.json` not committed yet | `bun run kata:manifest && git add kata-manifest.json` (first-time only) | | ATC missing from manifest after regen | Used template literal `` @atc(`PROJ-${id}`) `` instead of string literal | Change to `@atc('PROJ-XXX')` — the scanner only matches string literals | | Phantom ATC in manifest | `@atc(...)` example inside a JSDoc/comment was captured | Confirm the scanner is comment-aware (commit `c339533` fixed this); ensure the comment line begins with `//` or `*` | | Component missing from manifest | File listed in `EXCLUDED_FILES` (`scripts/kata-manifest.ts`) | Rename the file, or remove it from the exclusion list | | Class name in manifest looks wrong | First `export class PascalCase` in the file is not the intended one | Make the intended class the first export; or refactor the file | | Husky gate fires on unrelated commit | Staged files include `tests/components/**` (e.g. README inside the dir) | Move the unrelated file out of `tests/components/`, or accept the gate run | --- ## 8. Gotchas 1. **Dependency projects do not re-run between test projects.** `ui-setup` runs once per invocation. If you change auth credentials mid-session, invalidate `.auth/` manually (`rm -rf .auth`). 2. **Serial today does not license shared state.** The shipped config runs `fullyParallel: false` + `workers: 1`, but a test that would fail under parallelism has shared state — locate it and remove it now; do not reach for `test.describe.serial`, and do not let serial execution hide the bug that blocks the future workers bump. 3. **Project `testMatch` is case-sensitive on Linux, case-insensitive on macOS.** CI is Linux. Match the pattern exactly on disk. 4. **The `global-teardown` PROJECT runs even if all tests were skipped.** It is wired via the `teardown:` property on the `global-setup` project (not a `globalTeardown` hook). Use it for artifact cleanup and the run summary — but NOT for the TMS sync: it finishes before `KataReporter.onEnd()` writes `reports/atc_results.json`, so the write-back is a separate `bun run test:sync` step after the process exits. 5. **The `baseURL` applies to `page.goto('/path')` only.** API requests go through `ApiBase` which uses `config.apiUrl` from `config/variables.ts`. They are independent. 6. **Reporter order is preserved.** Moving `html` before `KataReporter` can cause the tree view to miss step events on fast tests. Keep `KataReporter` first. 7. **`retries: 0` applies in CI too.** There is no retry allowance anywhere in this config. A test that would only pass on retry is a bug — fix the test or the product, never the retry count. 8. **Setup projects produce side effects** (`.auth/*.json`, `reports/.atc_partial.ndjson`). Do not commit these. `.gitignore` covers them; do not weaken it. --- ## 9. Troubleshooting | Symptom | Likely cause | Fix | |---------|--------------|-----| | Tests pass locally, fail in CI with timeout | CI runner slower than local; insufficient waits | Add condition-based waits; do not bump the global timeout | | "forbidOnly" failure in CI | Left a `test.only` in the diff | Remove it; CI is protecting `main` | | Auth state missing in CI | Setup project did not run | Check `dependencies` chain; verify `.auth/` is not pre-existing from a previous run in cached workspaces | | Allure report empty | `allure-results/` not produced | Check `allure-playwright` is registered in `reporter`; verify `outputFolder` option | | Different browser runs different test set | One project has `testMatch` the other does not | Align `testDir` and `testMatch` per project | | `kata-manifest.json` missing an ATC | `@atc` uses a template literal or variable | Decorator scanner requires a string literal — `@atc('PROJ-101')` | | Storage state rejected — "token expired" | `.auth/api-state.json` stale | Delete `.auth/`, rerun; or shorten setup token TTL handling | | Tests that pass on retry | Hidden race condition | Fix the race; do not rely on `retries` | --- ## 10. Out of scope — handoff to `regression-testing` The following belong to the regression-testing skill, not here: - GitHub Actions workflow YAML (PR / main / nightly / release triggers) - `gh run list`, `gh run watch`, `gh run view --log-failed` - Allure report hosting / publishing pipelines - Slack / email failure notifications - Regression GO/NO-GO analysis - Flakiness classification across runs - Test sharding matrices for nightly suites - Secret management beyond "names and purposes of variables the config reads" If the task at hand is "run the regression suite in CI and analyse the result", start at `regression-testing`. If it is "configure Playwright so regression can run it consistently", stay here. -
data-testid-strategy.md 8.2 KB
# Data-testid Strategy for KATA + Playwright How to select elements in Playwright tests that live inside KATA Page components. Load when writing UI ATCs, reviewing brittle locators, or deciding what to do when the app is missing testids. --- ## 1. Locator priority (strict) Always pick the highest-priority option that works: | Priority | Locator | When to use | |----------|---------|-------------| | 1 | `page.getByTestId('name')` | Always preferred when a `data-testid` exists | | 2 | `page.getByRole(role, { name })` | Semantic elements (button, link, heading) with unambiguous accessible name | | 3 | `page.getByLabel('label text')` | Inputs with a real associated `<label>` | | 4 | `page.getByText('unique text')` | Unique visible content that the test's purpose requires | | 5 | CSS / XPath | Last resort only. Flag in review. | If a `data-testid` exists on the target, use it. Period. It is the most stable selector and the most resilient to design / copy / structure changes. --- ## 2. Naming conventions When reading the app code, expect this pattern (and when asking the dev team for new testids, request this pattern): | Context | Convention | Example | |---------|-----------|---------| | Component root | `camelCase` | `data-testid="shoppingCart"` | | Specific element | `snake_case` | `data-testid="email_input"` | | Component section | `snake_case` | `data-testid="billing_section"` | | Action button | `snake_case` | `data-testid="checkout_button"` | Pattern: `{description}_{type}` where `type` ∈ `input`, `button`, `link`, `section`, `list`, `item`, `modal`, `toast`, `badge`. For repeated elements in lists, put the testid on the **container** with the same name for each item; use `.nth(i)` or `.filter()` for specific ones. Do not embed dynamic IDs into the testid (`product_123`) unless that's the *only* way the app distinguishes them. --- ## 3. Syntax — getByTestId vs locator ```typescript // Preferred const loginButton = this.page.getByTestId('login_submit_button'); // Equivalent (older syntax, still valid) const loginButton = this.page.locator('[data-testid="login_submit_button"]'); // Wrong — fragile, breaks on every CSS refactor const loginButton = this.page.locator('.btn.btn-primary'); ``` `getByTestId` uses the `testIdAttribute` configured in `playwright.config.ts` (`data-testid` by default). If the project uses a non-default attribute (`data-qa`, `data-test`), configure it once in `playwright.config.ts` and keep using `getByTestId`. --- ## 4. KATA integration — where locators live | Case | Where to put the locator | Why | |------|-------------------------|-----| | Used in 2+ ATCs within the same Page | Private property on the Page component (lazy function form) | DRY within the component; single source of truth for that element | | Used in only 1 ATC | Inline inside the ATC | No premature abstraction; easier to read | | Used across multiple Page components | Still inline or on each Page — do NOT lift to UiBase | A shared locator means two pages share layout, which is usually wrong | Lazy function form prevents "target closed" errors when the page reloads mid-ATC: ```typescript // tests/components/ui/LoginPage.ts export class LoginPage extends UiBase { private readonly emailInput = () => this.page.getByTestId('email_input'); private readonly passwordInput = () => this.page.getByTestId('password_input'); private readonly submitButton = () => this.page.getByTestId('login_submit_button'); private readonly errorMessage = () => this.page.getByTestId('login_error_message'); @atc('AUTH-UI-001') async loginSuccessfully(email: string, password: string) { await this.emailInput().fill(email); await this.passwordInput().fill(password); await this.submitButton().click(); await expect(this.page).toHaveURL(/.*\/dashboard.*/); } @atc('AUTH-UI-002') async loginWithInvalidCredentials(email: string, password: string) { await this.emailInput().fill(email); await this.passwordInput().fill(password); await this.submitButton().click(); await expect(this.errorMessage()).toBeVisible(); } } ``` For locators used in only one ATC, keep them inline — do not lift: ```typescript @atc('ACCOUNT-UI-010') async deleteAccount(confirmation: string) { await this.page.getByTestId('account_delete_button').click(); await this.page.getByTestId('delete_confirmation_input').fill(confirmation); await this.page.getByTestId('delete_confirm_button').click(); await expect(this.page).toHaveURL(/.*\/goodbye.*/); } ``` --- ## 5. Common patterns ### Lists ```typescript const cards = this.page.getByTestId('product_card'); const count = await cards.count(); const firstCard = cards.first(); const lastCard = cards.last(); const thirdCard = cards.nth(2); for (const card of await cards.all()) { // iterate } ``` ### Filter by visible child text ```typescript const targetRow = this.page .getByTestId('user_row') .filter({ hasText: userEmail }); await targetRow.getByTestId('edit_button').click(); ``` ### Dynamic IDs in the testid When the app embeds an ID (`edit_product_123`), prefer a container-plus-filter pattern: ```typescript // Better: match pattern via RegExp const button = this.page.getByTestId(/^edit_product_/); // Or: scope by a parent container with a stable testid await this.page .getByTestId('product_card') .filter({ hasText: productName }) .getByTestId('edit_button') .click(); ``` ### Element states ```typescript // Loading await expect(this.page.getByTestId('loading_spinner')).toBeVisible(); await expect(this.page.getByTestId('loading_spinner')).toBeHidden(); // Empty state await expect(this.page.getByTestId('empty_state_message')).toHaveText('No results found'); // Error toast await expect(this.page.getByTestId('error_toast')).toContainText(/validation failed/i); ``` --- ## 6. When the app is missing testids Order of escalation: 1. **Check the design system.** Often testids are on the primitive component, not the app-level wrapper. `page.getByTestId('Button')` sometimes works surprisingly often. 2. **Try priorities 2-4** (role / label / unique text) first. Prefer `getByRole` with an accessible name; it's resilient and also validates a11y. 3. **File a request** with the frontend team. Include the screen, the element, the proposed testid name, and the ATC that needs it. Link to `data-testid-strategy.md` §2 for naming. 4. **Temporary workaround**: use `getByRole` + `getByText` scoped by a stable parent. Mark the ATC with `// TODO: data-testid needed for <desc>` and add the TODO to the PBI's `context.md`. Never ship a CSS-class-based selector as a permanent solution. They decay silently on every component refactor. --- ## 7. Anti-patterns (reject on review) | Anti-pattern | Example | Why it fails | |--------------|---------|--------------| | CSS class selector | `.locator('.btn-primary')` | Breaks on every design-system refactor | | Structural CSS | `.locator('div > div > button:nth-child(2)')` | Breaks on any DOM restructure, even cosmetic | | Hard-coded index on an unlabeled list | `.locator('button').nth(3)` | Fails when a new button is added | | XPath | `.locator('//div[@class="x"]/..//span')` | Brittle and unreadable | | Sharing a locator across Pages | `uiBase.loginButton` | Two Pages sharing a locator signals two Pages should be one, or one is wrong | | Inline literal string used in 4 ATCs | `.getByTestId('header_cart')` x4 | Extract to a Page property | --- ## 8. Debugging ```bash # UI mode — visualize locators and replay bun run test:ui # Debug mode — step through, inspect locators bun run test -- --debug # Codegen — record interactions and emit getByTestId bunx playwright codegen https://staging.example.com ``` Quick inventory of all testids on the current page (paste into DevTools): ```js [...document.querySelectorAll('[data-testid]')].map(e => e.getAttribute('data-testid')).sort() ``` Use this when you need to compare the ATP's assumed testids against the actual UI. --- ## 9. Configuration `playwright.config.ts`: ```typescript export default defineConfig({ use: { testIdAttribute: 'data-testid', // default; override if the project uses `data-qa` etc. }, }); ``` If the project uses a non-standard attribute, the skill still writes `page.getByTestId('...')` — only the config differs. -
e2e-patterns.md 19 KB
# E2E Patterns — Page Components, Locators, Waits > **Subagent context**: when the test-automation Code phase is dispatched (Sequential, see SKILL.md §Subagent Dispatch Strategy), this file is part of the subagent's "Context docs" briefing component. How to write a KATA `Page` component and E2E test that drives Playwright. Load when writing UI ATCs, picking locators, handling waits, or wiring API-assisted UI flows. Architecture, fixture selection, AAA, ATC naming, and test-file structure live in `kata-architecture.md` and `automation-standards.md` — this reference focuses on Playwright-specific mechanics. --- ## 1. Where UI code lives ``` tests/components/ui/ UiBase.ts # Layer 2 — Playwright helpers LoginPage.ts # Layer 3 — one file per page/feature CheckoutPage.ts tests/components/UiFixture.ts # registers every Page component tests/e2e/{module}/{verbFeature}.test.ts tests/setup/ui-auth.setup.ts # storageState generator (optional) ``` A Page component **is not** a 1:1 map of a URL. It is the smallest group of ATCs that share the same domain (login, checkout, product listing). Pages with more than ~15–20 ATCs should be split. --- ## 2. Page component skeleton ```typescript import type { TestContextOptions } from '@TestContext'; import { expect } from '@playwright/test'; import { UiBase } from '@ui/UiBase'; import { atc } from '@utils/decorators'; export interface LoginCredentials { email: string; password: string; } export class LoginPage extends UiBase { // 1. Shared locators (only when used in 2+ ATCs) private readonly emailInput = () => this.page.getByTestId('email_input'); private readonly passwordInput = () => this.page.getByTestId('password_input'); private readonly submitButton = () => this.page.getByTestId('login_submit_button'); // 2. Constructor constructor(options: TestContextOptions) { super(options); } // 3. Navigation helper async goto(): Promise<void> { await this.page.goto('/login'); await this.page.waitForLoadState('networkidle'); } // 4. ATCs @atc('TICKET-ID') async loginWithValidCredentials(data: LoginCredentials): Promise<void> { await this.emailInput().fill(data.email); await this.passwordInput().fill(data.password); await this.submitButton().click(); await expect(this.page).toHaveURL(/.*dashboard.*/); } @atc('TICKET-ID') async loginWithInvalidCredentials(data: LoginCredentials): Promise<void> { await this.emailInput().fill(data.email); await this.passwordInput().fill(data.password); await this.submitButton().click(); await expect(this.page.locator('[role="alert"]')).toBeVisible(); await expect(this.page).toHaveURL(/.*\/login.*/); } } ``` Method order inside the class: shared locators → constructor → navigation → ATCs. Private helpers at the bottom only if they are used 2+ times within the class. --- ## 3. Locator strategy ### Priority ladder | Priority | Strategy | Use when | |----------|---------|----------| | 1 | `getByTestId('X')` or `locator('[data-testid="X"]')` | Always preferred — contract between dev and QA | | 2 | `getByRole('button', { name: 'Submit' })` | Semantic elements with accessible name | | 3 | `getByLabel('Email')` | Form inputs with an associated `<label>` | | 4 | `getByText('Sign in')` | Unique visible text; fragile if copy changes | | 5 | CSS / XPath (`button[type="submit"]`) | Last resort | Never mix priorities in one selector (`.container [data-testid="x"]`). Never chain DOM structure (`div > form > button:nth-child(3)`). Never reach for `.btn-primary` or other class hooks — styling can change. ### Playwright selector syntax ```typescript // CORRECT — getByTestId (preferred) const loginButton = page.getByTestId('login_submit_button'); // CORRECT — locator with CSS attribute (also valid, same stability) const loginButton = page.locator('[data-testid="login_submit_button"]'); // CORRECT — role-based when no testid exists const loginButton = page.getByRole('button', { name: /submit/i }); // WRONG — CSS class (fragile, changes with styling) const loginButton = page.locator('.btn-primary'); // WRONG — DOM structure (fragile, changes with layout refactors) const loginButton = page.locator('div > form > button:last-child'); ``` ### `data-testid` naming contract (used in application code) The product code owns the contract; tests only read it. Expect: | Context | Convention | Pattern | Example | |---------|-----------|---------|---------| | Component root | camelCase | `{componentName}` | `data-testid="shoppingCart"` | | Text input | snake_case | `{description}_input` | `data-testid="email_input"` | | Button | snake_case | `{description}_button` | `data-testid="checkout_button"` | | Link | snake_case | `{description}_link` | `data-testid="forgot_password_link"` | | Section / container | snake_case | `{description}_section` | `data-testid="billing_section"` | | List container | snake_case | `{description}_list` | `data-testid="order_list"` | | List item | snake_case | `{description}_item` | `data-testid="order_item"` | | Error message | snake_case | `{description}_error` | `data-testid="form_email_error"` | | Loading state | snake_case | `{description}_loading` | `data-testid="products_loading"` | | Empty state | snake_case | `{description}_empty_state` | `data-testid="products_empty_state"` | General pattern: `{description}_{type}` where `type` identifies the element's role. Component roots use camelCase; all nested elements use snake_case. ### Locator anti-patterns ```typescript // WRONG: CSS class selectors — break when styling changes page.locator('.btn-primary'); // WRONG: DOM structure selectors — break when layout changes page.locator('div > form > button:last-child'); // WRONG: Text that may change (copy updates, i18n) page.getByText('Sign In'); // may change to 'Login' or 'Iniciar sesión' // WRONG: Hardcoded index without reason page.locator('[data-testid="card"]').nth(2); // why the third one? // WRONG: Mixing CSS with data-testid page.locator('.container [data-testid="button"]'); // RIGHT: Direct data-testid page.getByTestId('login_submit_button'); // RIGHT: Role-based for semantic elements without testid page.getByRole('button', { name: /submit/i }); // RIGHT: Filter by content when needed page.getByTestId('product_card').filter({ hasText: 'iPhone' }); // RIGHT: Specific dynamic testid page.getByTestId(`product_card_${productSlug}`); ``` ### When a `data-testid` is missing 1. **Verify it exists** — use browser DevTools: `document.querySelectorAll('[data-testid]')` or the testid enumeration snippet in section 10. 2. **File a ticket** against the application repo with: component path, route, element description, and a proposed name following the naming contract above. 3. **Work around temporarily** with `getByRole` or `getByLabel` and leave a `// TODO: replace with getByTestId('X') when DEV adds the testid` comment. 4. Never ship a brittle CSS/XPath fallback without the TODO — otherwise it hides the tech debt. ### Inline vs shared locators Locators default to **inline** inside the ATC. Extract to a `private readonly` arrow function on the class only when the **same** locator is used in 2+ ATCs of that component. ```typescript // inline (default) — used once await this.page.locator('[data-testid="forgot_password_link"]').click(); // shared — used in 2+ ATCs of this class private readonly cartTotal = () => this.page.locator('[data-testid="cart_total"]'); private readonly productRow = (name: string) => this.page.locator(`[data-product="${name}"]`); ``` Never put locators in a separate `locators/*.ts` file. Never wrap a single `page.fill()` or `page.click()` in a private helper. ### Lists and dynamic IDs ```typescript // Multiple elements with the same testid const cards = this.page.getByTestId('product_card'); await expect(cards).toHaveCount(3); const first = cards.nth(0); // Filter by text const iphoneCard = cards.filter({ hasText: 'iPhone' }); // Dynamic id prefix const editButton = this.page.locator('[data-testid^="edit_product_"]').first(); // Exact dynamic id await this.page.getByTestId(`edit_product_${productId}`).click(); ``` --- ## 4. Waits — no timeouts, condition-first Playwright auto-waits on locator actions (`click`, `fill`, `toBeVisible`). Explicit waits are only needed when timing depends on a specific network call, a navigation, or a DOM mutation that Playwright cannot infer. ### Allowed wait patterns ```typescript // Network-driven await this.page.waitForLoadState('networkidle'); // last resort — slow await this.page.waitForResponse(r => r.url().includes('/api/cart') && r.ok()); await this.page.waitForURL(u => !u.pathname.includes('/login')); // DOM-driven await this.page.waitForSelector('[data-loaded="true"]'); await expect(locator).toBeVisible(); await expect(locator).toHaveCount(expected); ``` ### Banned patterns ```typescript // NEVER await this.page.waitForTimeout(3000); // arbitrary, flaky await this.page.waitForTimeout(500); // even small values await setTimeout(...); // node-level sleep ``` Retries masks the same problem (`retries: 0` in `playwright.config.ts`). If a test passes on retry, it is flaky — investigate the root cause. ### Conditional UI (popups, modals) ```typescript const popup = this.page.locator('[role="dialog"]'); const isVisible = await popup.isVisible({ timeout: 2000 }).catch(() => false); if (isVisible) await popup.locator('button:has-text("Close")').click(); ``` ### Intercepting the real backend call ```typescript await Promise.all([ this.page.waitForResponse(r => r.url().includes('/api/cart')), this.page.locator('[data-testid="add_to_cart_button"]').click(), ]); ``` --- ## 5. UiBase helpers — interception UiBase exposes two wrappers around Playwright's response APIs. Use them when a UI action fires a network call whose payload or status code is part of the assertion. ### `interceptResponse` — capture response from an action ```typescript @atc('TICKET-ID') async loginAndCaptureToken(credentials: LoginCredentials) { await this.goto(); await this.emailInput().fill(credentials.email); await this.passwordInput().fill(credentials.password); const { responseBody, status } = await this.interceptResponse<LoginPayload, TokenResponse>({ urlPattern: /\/auth\/login/, action: async () => { await this.submitButton().click(); }, }); expect(status).toBe(200); expect(responseBody?.access_token).toBeDefined(); await this.page.waitForURL(url => !url.pathname.includes('/login')); } ``` ### `waitForApiResponse` — wait for an already-triggered response ```typescript @atc('TICKET-ID') async loadOrdersAndVerifyCount(): Promise<void> { await this.goto(); await this.page.locator('[data-testid="apply_filter_button"]').click(); const { responseBody } = await this.waitForApiResponse<void, Order[]>({ urlPattern: /\/api\/orders/, }); await expect(this.page.locator('[data-testid="order_item"]')).toHaveCount(responseBody?.length ?? 0); } ``` Intercepted payloads attach automatically to the Allure report. --- ## 6. Fixed assertions and test-level assertions Every UI ATC contains **fixed assertions** that validate the primary expected outcome: ```typescript @atc('TICKET-ID') async loginWithValidCredentials(data: LoginCredentials): Promise<void> { await this.emailInput().fill(data.email); await this.passwordInput().fill(data.password); await this.submitButton().click(); // Fixed assertions — live inside the ATC await this.page.waitForURL(u => !u.pathname.includes('/login')); await expect(this.page).not.toHaveURL(/.*\/login.*/); } ``` Additional, test-specific assertions go in the test file: ```typescript test('TICKET-ID: should show welcome banner after login', async ({ ui }) => { await ui.login.loginWithValidCredentials(credentials); await expect(ui.page.locator('[data-testid="welcome_message"]')).toContainText('Welcome'); }); ``` --- ## 7. Test file template (E2E) ```typescript import { test, expect } from '@TestFixture'; test.describe('TICKET-ID: Validate checkout flow', () => { let product: ProductCandidate | null; test.beforeAll(async ({ api }) => { // DISCOVER — no assertions (see test-data-management.md) product = await api.products.findAvailableProduct(); }); test('TICKET-ID: should complete checkout when cart is valid @critical', async ({ test: fixture }) => { if (!product) return test.skip(true, 'No available product'); const { api, ui } = fixture; // ARRANGE — API setup, UI action, API verification await api.auth.loginSuccessfully(credentials); const checkoutData = ui.data.createCheckoutData(); // ACT await ui.checkout.goto(); await ui.checkout.completeCheckoutSuccessfully(checkoutData); // ASSERT (beyond the ATC's fixed assertions) const [, orders] = await api.orders.getOrdersSuccessfully({ customerId: product.customerId }); expect(orders.length).toBeGreaterThan(0); }); test('TICKET-ID: should show error with invalid email', async ({ ui }) => { if (!product) return test.skip(true, 'No available product'); const invalid = ui.data.createCheckoutData({ email: 'invalid-email' }); await ui.checkout.goto(); await ui.checkout.completeCheckoutWithInvalidEmail(invalid); }); }); ``` Rules applied in the skeleton above (all mandatory): - `test` imported from `@TestFixture` (not `@playwright/test`). - Ticket ID prefix in both `describe` and `test`. - `beforeAll` discovers data without assertions; each test guards with `test.skip()` (see `test-data-management.md` §7). - Dynamic data via `ui.data.*` / `api.data.*`, never hardcoded. - Tags (`@critical`, `@smoke`, `@regression`) applied at the correct level. --- ## 8. Hybrid testing — API setup + UI flow The fastest reliable pattern: set up state via API (no browser UI fiddling), drive the feature via UI, verify via API. ```typescript test('TICKET-ID: should create order via UI and verify via API', async ({ test: fixture }) => { const { api, ui } = fixture; await api.auth.loginSuccessfully({ email, password }); await ui.orders.goto(); await ui.orders.createOrderSuccessfully({ customerId: 123, guestEmail: ui.generateEmail('order-test'), confirmationNumber: ui.faker.string.alphanumeric(10), }); const [, orders] = await api.orders.getOrdersSuccessfully({ customerId: 123 }); expect(orders.length).toBeGreaterThan(0); }); ``` Pick the fixture by test shape (`automation-standards.md` §1 has the full matrix): `{ui}` for pure UI, `{test}` for hybrid, `{api}` for API-only. --- ## 9. Authenticated tests — storage state reuse For suites where login is a precondition for most tests, log in once and reuse the resulting storage state. ```typescript // tests/setup/ui-auth.setup.ts import { expect, test as setup } from '@playwright/test'; import { config } from '@variables'; const authFile = 'playwright/.auth/user.json'; setup('authenticate', async ({ page, request }) => { const response = await request.post(`${config.apiUrl}/auth/login`, { data: { email: config.testUser.email, password: config.testUser.password }, }); expect(response.ok()).toBeTruthy(); const { access_token } = await response.json(); await page.goto(config.baseUrl); await page.evaluate(token => localStorage.setItem('token', token), access_token); await page.context().storageState({ path: authFile }); }); ``` ```typescript // playwright.config.ts (excerpt) projects: [ { name: 'setup', testMatch: /.*\.setup\.ts/ }, { name: 'e2e', dependencies: ['setup'], use: { storageState: 'playwright/.auth/user.json' }, }, ], ``` Use API login (not UI login) inside the setup — faster and less fragile. For multi-role test runs, generate one storageState per role. --- ## 10. Common UI patterns ### Modal / dialog ```typescript await expect(this.page.locator('[data-testid="confirm_modal"]')).toBeVisible(); await this.page.locator('[data-testid="confirm_button"]').click(); await expect(this.page.locator('[data-testid="confirm_modal"]')).not.toBeVisible(); ``` ### Lists with dynamic count ```typescript await this.page.waitForSelector('[data-testid="item_list"] [data-testid="item"]'); const items = this.page.locator('[data-testid="item_list"] [data-testid="item"]'); await expect(items).toHaveCount(expectedCount); ``` ### Multi-step form ```typescript await this.page.locator('[data-testid="step_1_input"]').fill(data.step1Value); await this.page.locator('[data-testid="next_button"]').click(); await expect(this.page.locator('[data-testid="step_2_section"]')).toBeVisible(); await this.page.locator('[data-testid="step_2_input"]').fill(data.step2Value); await this.page.locator('[data-testid="submit_button"]').click(); ``` ### Element states (loading / empty / error) ```typescript await expect(this.page.getByTestId('products_loading')).toBeVisible(); await expect(this.page.getByTestId('products_empty_state')).toContainText('No products found'); await expect(this.page.getByTestId('products_error_state')).toBeVisible(); ``` ### Form field errors ```typescript await expect(this.page.getByTestId('form_name_error')).toHaveText('Name is required'); await expect(this.page.getByTestId('form_email_error')).toHaveText('Invalid email'); ``` ### Debugging: enumerate all testids on a page ```typescript test('dev-only: list testids on /login', async ({ page }) => { await page.goto('/login'); const ids = await page.evaluate(() => Array.from(document.querySelectorAll('[data-testid]')).map(el => ({ testId: el.getAttribute('data-testid'), tag: el.tagName, text: el.textContent?.slice(0, 50), })), ); console.table(ids); }); ``` Use `playwright test --ui` (inspector) or `--debug` to step through locator resolution. --- ## 11. Running E2E tests ```bash # Whole suite bun run test:e2e # One file bun run test tests/e2e/auth/login.test.ts # By tag bun run test:e2e --grep @critical bun run test:e2e --grep @smoke # By browser (if multi-project configured) bun run test:e2e --project=chromium bun run test:e2e --project=firefox # Interactive debugging bun run test:ui bun run test --debug tests/e2e/auth/login.test.ts bun run test --trace on tests/e2e/auth/login.test.ts # Report bun run allure:generate ``` --- ## 12. Implementation checklist (E2E coding phase) Before leaving the coding phase and running the review: - [ ] Component extends `UiBase`, constructor takes `TestContextOptions` and calls `super`. - [ ] Every ATC decorated with `@atc('TICKET-ID')` where the ID matches a real TMS case. - [ ] Locators inline unless used in 2+ ATCs (then `private readonly` arrow). - [ ] Locator priority respected (`getByTestId` first). - [ ] Every ATC contains at least one fixed assertion. - [ ] No `waitForTimeout`. - [ ] New component registered in `UiFixture.ts`. - [ ] Test file under `tests/e2e/{module}/`, name is `{verb}{Feature}.test.ts`. - [ ] `test` imported from `@TestFixture`. - [ ] Ticket ID prefix in describe/test. - [ ] Data via `ui.data.*` / `api.data.*`, no hardcoding. - [ ] `beforeAll` has no assertions; each test uses `test.skip()` guard. - [ ] Tags (`@critical`, `@smoke`, `@regression`) applied. - [ ] `bun run test <file>` passes. - [ ] `bun run types:check` clean. - [ ] `bun run lint:check` clean. -
explain-tests.md 2.8 KB
# Test Execution Breakdown Generate a plain-English breakdown of what automated tests do: which ATCs run, what assertions fire, and how data flows. **Input:** $ARGUMENTS (Scope: a file path, ATC ID, ticket ID, or module name. Examples: `tests/e2e/dashboard/dashboard.test.ts`, `AUTH-003`, `AUTH-T01`, `all auth tests`.) --- ## Step 1: Identify Scope Parse `$ARGUMENTS` to determine the scope: | Input looks like | Scope | Action | |-----------------|-------|--------| | File path (`.test.ts`) | Single test file | Read that file | | ATC ID (e.g. `AUTH-003`) | Single ATC | Find the ATC method in `tests/components/` | | Ticket ID (e.g. `AUTH-T01`) | Ticket tests | Find all test files referencing that ticket | | Module name (e.g. `auth`) | Full module | Find all test files under that module folder | Read the actual source code for the identified scope. Never guess what assertions exist. ## Step 2: Trace the Code For each test file in scope: 1. **Read the test file** to identify test blocks (`test()`, `test.describe()`) 2. **Read each ATC** referenced by the tests (the component methods in `tests/components/`) 3. **Read any Steps** used as preconditions 4. **Note fixture usage** (`{ api }`, `{ ui }`, `{ test }`) ## Step 3: Generate the Breakdown ### Setup Section (if applicable) Show shared setup (beforeAll, beforeEach, discovery, fixtures): ``` SETUP: {description} +-- {what it does} | +-- {how it does it} +-- RESULT: | +-- variable1 = { discovered/computed value } | +-- variable2 = { discovered/computed value } +-- Reused in: {list of tests} ``` ### Per-Test Section For each test: ``` TEST: "{test name}" GUARD: {skip condition, if any} FIXTURE: { api | ui | test } ATC {ID}: {methodName}({ parameters }) | +-- ACTION: | +-- {step 1} | +-- {step 2} | +-- POSITIVE ASSERTIONS (must be true): | +-- {assertion 1} | +-- {assertion 2} | +-- NEGATIVE ASSERTIONS (must NOT be true): +-- {assertion 1} +-- {assertion 2} VALIDATES: {one sentence -- business value of this test} ``` For multi-ATC tests, show each ATC as a separate numbered step. For parameterized tests, show the data table and which partition each row covers. ### Summary Table | Test | ATC(s) | Assertions | What It Validates | |------|--------|------------|-------------------| | ... | ... | {count} | {business description} | ### Reused Variables Table (if setup discovered data) | Variable | Source | Used In | |----------|--------|---------| | ... | ... | ... | ## Rules 1. Read the actual code before generating -- never invent assertions 2. Count assertions accurately per ATC and per test 3. Group assertions by category (positive, negative, structural) 4. Explain in business terms what each test validates 5. Keep the output in English, formatted for documentation or PR descriptions -
kata-architecture.md 23.2 KB
# KATA Architecture — Layers, Fixtures, ATCs, Steps Full reference for the Komponent Action Test Architecture (KATA). Load when designing new components, picking fixtures, wiring ATCs, or building Steps chains. > **Canonical formula — the one sentence every KATA surface must agree with:** > > **KATA organises automation in four layers with a single direction of dependency: TestContext, > Base, domain Components and Fixtures. Steps is an optional intermediate layer between Components > and Fixtures. Test files consume the Fixtures — they are not a layer.** > > Short form, for a chip or a title: **"four named layers, plus optional Steps"**. Never publish a > bare number. Banned on every surface: "three layers", "five layers", "6 layers", "Test files" as > a layer, and the acronym expanded with a C ("Component Action Test Architecture"). The expansion is > **Komponent Action Test Architecture**: the K lives only in the name; the layer is "domain > Components", spelled normally. DRY zones that are NOT layers: `tests/utils/`, `tests/data/`, > `config/`. Consumers: `tests/e2e/`, `tests/integration/`. --- ## 1. The Four Layers ``` Layer 4 — Fixtures (DI) TestFixture, ApiFixture, UiFixture | v Layer 3.5 — Steps (optional) AuthSteps, CheckoutSteps — reusable ATC chains for preconditions | v Layer 3 — Domain Components (ATCs) UsersApi, LoginPage, CheckoutPage — @atc lives here | v Layer 2 — Base Components ApiBase (HTTP helpers), UiBase (Playwright helpers) | v Layer 1 — TestContext Config, logger, faker, environment accessors ``` | Layer | Responsibility | Typical files | |-------|---------------|---------------| | 1 — TestContext | Global utilities (config, logger, faker, env) | `TestContext.ts` | | 2 — Base | HTTP and Playwright helpers | `ApiBase.ts`, `UiBase.ts` | | 3 — Domain | Business logic, ATCs | `UsersApi.ts`, `LoginPage.ts` | | 3.5 — Steps | Reusable ATC chains for preconditions | `AuthSteps.ts`, `CheckoutSteps.ts` | | 4 — Fixtures | DI entry point | `TestFixture.ts`, `ApiFixture.ts`, `UiFixture.ts` | | Tests | Orchestrate ATCs into scenarios | `tests/e2e/**`, `tests/integration/**` | Rule: a component in a higher layer may use a lower layer, never the other way round. --- ## 2. Directory Structure ``` /config variables.ts # Single source for env vars + URLs /tests KataReporter.ts # Custom Playwright reporter /components TestContext.ts # Layer 1 ApiFixture.ts # Layer 4 (API only) UiFixture.ts # Layer 4 (UI only) TestFixture.ts # Layer 4 (unified: api + ui) /api ApiBase.ts # Layer 2 UsersApi.ts # Layer 3 /ui UiBase.ts # Layer 2 LoginPage.ts # Layer 3 /steps ExampleSteps.ts # Layer 3.5 (reference pattern) /data DataFactory.ts # Typed faker-backed factories types.ts # Payload / domain TypeScript types constants.ts # Static test values, boundaries (create-on-demand) /fixtures # Static JSON/CSV rows /mocks # Canned mock/stub responses (create-on-demand) /integration/{module} # API-only tests /e2e/{module} # UI (+API) tests /setup global.setup.ts # Global setup (+ api-auth / ui-auth setup projects) /teardown global.teardown.ts # Global teardown /utils decorators.ts # @atc, @step ``` `tests/data/` canonical TypeScript files (naming is fixed; create only when the scope needs them): | File | Holds | Example members | |------|-------|-----------------| | `DataFactory.ts` | Typed, faker-backed factories that build payloads at runtime | `generateUserPayload()`, `generateOrderPayload()` | | `types.ts` | Payload / domain TypeScript types shared by factories and ATCs | `UserPayload`, `OrderPayload` | | `constants.ts` | Static test values, magic numbers, boundary constants | `MAX_ORDER_ITEMS`, `DEFAULT_PAGE_SIZE` | These three complement the static data folders: `tests/data/fixtures/` (parameterization rows as JSON/CSV) and `tests/data/mocks/` (canned mock/stub responses). Factories generate fresh runtime data; fixtures and mocks hold committed static data. Naming + folder conventions for those two live in `automation-standards.md` §6. Import aliases are mandatory (no relative imports): ```typescript import { config, env } from '@variables'; import { ApiBase } from '@api/ApiBase'; import { LoginPage } from '@ui/LoginPage'; import { atc, step } from '@utils/decorators'; import { TestContext, type TestContextOptions } from '@TestContext'; ``` --- ## 3. TestContext (Layer 1) Stores Playwright drivers and global utilities. Every higher layer extends it or receives its options. ```typescript interface TestContextOptions { page?: Page; // Optional — API tests don't need it request?: APIRequestContext; // Optional — some setups don't need it environment?: Environment; } ``` Responsibilities: select URLs/credentials from `config` based on `TEST_ENV`; expose `faker`; expose environment label; provide shared logger/attachment helpers. Do not put Playwright-specific logic here (belongs in UiBase). Do not put HTTP-specific logic here (belongs in ApiBase). --- ## 4. Base Components (Layer 2) ### ApiBase Wraps `APIRequestContext` with type-safe HTTP helpers that return tuples: ```typescript protected async apiGET<T>(path: string): Promise<[APIResponse, T]> protected async apiPOST<T, P>(path: string, payload: P): Promise<[APIResponse, T, P]> protected async apiPUT<T, P>(path: string, payload: P): Promise<[APIResponse, T, P]> protected async apiPATCH<T, P>(path: string, payload: P): Promise<[APIResponse, T, P]> protected async apiPOSTForm<T>(path: string, form: FormData, options?: RequestOptions): Promise<[APIResponse, T]> protected async apiDELETE<T>(path: string): Promise<[APIResponse, T]> ``` ### UiBase Extends `TestContext`. Adds Playwright helpers: response interception, network waiting, storage-state snapshots, Allure attachments. Rule: anything that needs `PageContext` goes in UiBase; anything needing `APIRequestContext` goes in ApiBase; anything agnostic (allure attachments, string helpers) goes in `tests/utils/`. --- ## 5. Domain Components (Layer 3) One component per file. Max 15–20 ATCs per component — split if larger. | Type | Class | File | |------|-------|------| | API | `{Resource}Api` | `{Resource}Api.ts` | | UI | `{Page}Page` | `{Page}Page.ts` | Order inside the class: constructor → navigation (UI only) → helpers (no decorator or `@step`) → ATCs (`@atc`). ### API component template ```typescript import { expect, type APIResponse } from '@playwright/test'; import { ApiBase } from '@api/ApiBase'; import { atc, step } from '@utils/decorators'; import type { TestContextOptions } from '@TestContext'; export interface UserPayload { name: string; email: string; } export interface UserResponse { id: string; name: string; email: string; } export class UsersApi extends ApiBase { constructor(options: TestContextOptions) { super(options); } @step async getUserById(id: string): Promise<[APIResponse, UserResponse]> { return this.apiGET<UserResponse>(`/users/${id}`); } @atc('TICKET-ID') async createUserSuccessfully(payload: UserPayload): Promise<[APIResponse, UserResponse, UserPayload]> { const [response, body, sent] = await this.apiPOST<UserResponse, UserPayload>('/users', payload); expect(response.status()).toBe(201); expect(body.id).toBeDefined(); return [response, body, sent]; } } ``` ### UI component template ```typescript import { expect } from '@playwright/test'; import { UiBase } from '@ui/UiBase'; import { atc } from '@utils/decorators'; import type { TestContextOptions } from '@TestContext'; export interface LoginData { email: string; password: string; } export class LoginPage extends UiBase { private readonly submitButton = () => this.page.locator('button[type="submit"]'); constructor(options: TestContextOptions) { super(options); } @atc('TICKET-ID') async loginWithValidCredentials(data: LoginData): Promise<void> { await this.page.goto('/login'); await this.page.locator('#email').fill(data.email); await this.page.locator('#password').fill(data.password); await this.submitButton().click(); await expect(this.page).toHaveURL(/.*dashboard.*/); } } ``` --- ## 6. ATC Rules An ATC (Acceptance Test Case) is a **complete test case (mini-flow), not a single interaction**. Each ATC maps 1:1 to a ticket via `@atc('TICKET-ID')`. ### Rule 1 — Complete flow ```typescript // WRONG — single interaction @atc('TICKET-ID') async clickLoginButton() { await this.page.click('#login'); } // RIGHT — complete mini-flow @atc('TICKET-ID') async loginWithValidCredentials(data: LoginData) { await this.page.goto('/login'); await this.page.locator('#email').fill(data.email); await this.page.locator('#password').fill(data.password); await this.page.locator('button[type="submit"]').click(); await expect(this.page).toHaveURL(/.*dashboard.*/); } ``` ### Rule 2 — TC Identity = Precondition + Action A TC is defined by exactly two elements: the precondition (state) and the action (trigger). Every expected result from the same precondition + action is an assertion of the **same TC**, no matter which panel or endpoint it validates. ``` // WRONG — three TCs share the same precondition and action TC-A: Open published product -> verify Pricing block TC-B: Open published product -> verify Reviews block TC-C: Open published product -> verify page structure // RIGHT — one TC with all assertions TC: Open product detail page for published in-stock product Precondition: product is published and stock > 0 Action: user navigates to the product detail page Expected: - Page structure visible - Pricing values correct (Base - Discount = Final) - Inventory metrics correct - Reviews block shows rating + count - Add to Cart enabled ``` A TC is only different when the **precondition** or **action** changes — not when you check a different field of the same response. ### Rule 3 — Equivalence Partitioning Same expected output = one parameterized ATC. ```typescript // WRONG — three ATCs, same output (HTTP 401) @atc('T1') async loginWithWrongEmail() {} @atc('T2') async loginWithWrongPassword() {} @atc('T3') async loginWithEmptyFields() {} // RIGHT — one parameterized ATC @atc('T1') async loginWithInvalidCredentials(payload: LoginPayload) { // Test file parameterizes different invalid inputs; all produce 401 } ``` Different status codes, different UI states, or different business outcomes = separate ATCs. Same outcome, different data = one parameterized ATC. Minor conditional variations within the same behavior are acceptable; fundamentally different behavior is a separate ATC. > Rule 3 is a *within-partition* dedup rule — it does NOT authorize collapsing distinct partitions, boundaries, or states into one ATC, and it does NOT replace Boundary Value Analysis. One AC still maps to multiple ATCs (1:N): one per partition + boundary cases + state transitions. Derivation canon + triggers: `agentic-qa-core/references/test-design-doctrine.md`. ### Rule 4 — Locators inline Locators go inside the ATC, not in separate files. ```typescript // WRONG export const LOCATORS = { email: '#email', password: '#password' }; // RIGHT @atc('TICKET-ID') async loginWithValidCredentials(data: LoginData) { await this.page.locator('#email').fill(data.email); await this.page.locator('#password').fill(data.password); } ``` Exception — if the same locator is used in 2+ ATCs of the same component, extract to a `private readonly` arrow function on the class: ```typescript class CheckoutPage extends UiBase { private readonly cartTotal = () => this.page.locator('[data-testid="cart-total"]'); private readonly productRow = (name: string) => this.page.locator(`[data-product="${name}"]`); } ``` Never extract a locator used once. Never wrap a single `page.fill()` or `page.click()` in a helper. ### Rule 5 — ATCs do not call ATCs ATCs are atomic. For reusable chains across multiple test files, use the Steps module (Section 8). ```typescript // WRONG @atc('TICKET-ID') async checkoutWithNewUser() { await this.signupSuccessfully(userData); // calling an ATC! await this.addToCartSuccessfully(product); } // RIGHT — test file orchestrates, or Steps module test('TICKET-ID: should checkout with new user', async ({ ui }) => { await ui.signup.signupWithValidCredentials(userData); await ui.cart.addToCartSuccessfully(product); }); ``` ### Rule 6 — Fixed assertions inside, flow assertions outside Fixed assertions (status code, required fields, URL redirect) go inside the ATC. Test-level assertions (business outcomes combining multiple ATCs) go in the test file. ```typescript @atc('TICKET-ID') async signInSuccessfully(payload: SignInPayload) { const [response, body] = await this.apiPOST<AuthResponse, SignInPayload>('/auth/signin', payload); expect(response.status()).toBe(200); // FIXED expect(body.session.access_token).toBeDefined(); // FIXED return [response, body]; } // test file test('TICKET-ID: should persist session after login', async ({ test }) => { const [, body] = await test.api.auth.signInSuccessfully(credentials); expect(body.session.access_token).toMatch(/^eyJ/); // TEST-LEVEL }); ``` ### Rule 7 — Helpers vs ATCs | Type | What it does | `@atc` | |------|-------------|--------| | Helper | Retrieves data (read-only, no state change) | No — optional `@step` for tracing | | ATC | Performs an action that changes state | Yes — `@atc('TICKET-ID')` | A GET that validates access control (403/401) is still a helper — the ATC is the action that established the context. The GET belongs *inside* the ATC as a verification step. ```typescript // WRONG — bare GET as an ATC @atc('TICKET-ID') async getCurrentUserUnauthorized() { ... } // RIGHT — GET embedded inside the real action @atc('TICKET-ID') async loginWithInvalidCredentials(credentials: LoginPayload) { const [loginResp] = await this.apiPOST('/auth/login', credentials); const [meResp] = await this.apiGET('/auth/me'); expect(loginResp.status()).toBe(401); expect(meResp.status()).toBe(401); } ``` ### ATC structure ``` 1. Preconditions -> received via parameters (not internal setup) 2. Action -> POST / PUT / click / submit 3. Verification -> optional GET or page check 4. Assertions -> fixed assertions on expected outcome 5. Return -> [response, body, payload] for API, void for UI ``` ### ATC naming Format: `{verb}{Resource}{Scenario}` | Scenario | Suffix | Example | |----------|--------|---------| | Success | `Successfully` / `WithValidCredentials` | `createOrderSuccessfully` | | Invalid input | `WithInvalid{X}` | `loginWithInvalidCredentials` | | Not found | `WithNonExistent{X}` | `getUserWithNonExistentId` | | Expired | `WithExpired{X}` | `loginWithExpiredToken` | --- ## 7. Fixtures (Layer 4) Fixtures wire components together and expose them to tests. Three kinds (see `tests/components/TestFixture.ts` — the source of truth): | Fixture | Exposes | Opens browser? | |---------|---------|----------------| | `ApiFixture` | `api.users`, `api.orders`, ... | No | | `UiFixture` | `ui.login`, `ui.checkout`, ... | Yes | | `TestFixture` | `test.api` + `test.ui` in one object (shared context) | Yes | Steps classes are NOT a fixture — they live in `tests/components/steps/` and are instantiated directly (see §8). ### TestFixture definition The real file (`tests/components/TestFixture.ts`) extends Playwright's `test` with the three fixtures. Simplified shape: ```typescript import { test as base } from '@playwright/test'; import { ApiFixture } from '@ApiFixture'; import { UiFixture } from '@UiFixture'; export const test = base.extend<{ test: TestFixture; // TestFixture extends TestContext, holds .api + .ui api: ApiFixture; ui: UiFixture; }>({ test: async ({ page, request }, use) => { await use(new TestFixture(page, request)); }, api: async ({ request }, use) => { await use(new ApiFixture({ request })); }, ui: async ({ page, request }, use) => { await use(new UiFixture({ page, request })); }, }); export { expect } from '@playwright/test'; ``` ### Lazy initialisation Playwright only instantiates the fixtures a test requests. API-only tests never open a browser. ```typescript // No browser test('TICKET-ID: should get orders', async ({ api }) => { await api.orders.getOrders({ limit: 10 }); }); // Browser opens test('TICKET-ID: should view order list', async ({ ui }) => { await ui.orders.navigateTo(); }); // Browser opens, API and UI share the same context test('TICKET-ID: should create via API and verify via UI', async ({ test }) => { const [, order] = await test.api.orders.createOrderSuccessfully(data); await test.ui.orders.verifyOrderVisibleInList(order.id); }); ``` ### Fixture selection table | Test type | Fixture | Browser? | Use when | |-----------|---------|----------|----------| | API only (integration) | `{ api }` | No | Pure API testing. Default for `tests/integration/**`. | | UI only | `{ ui }` | Yes | UI-focused testing, no API setup. | | Hybrid | `{ test }` | Yes | API setup + UI action + API verification. | Reusable precondition chains (3+ ATCs repeated across 3+ files) are NOT a fixture — they go in a Steps class (§8), instantiated directly in the test. ### Registration pattern ```typescript // tests/components/UiFixture.ts export class UiFixture extends TestContext { readonly login: LoginPage; readonly checkout: CheckoutPage; constructor(options: TestContextOptions) { super(options); this.login = new LoginPage(options); this.checkout = new CheckoutPage(options); } } ``` Creating a component without registering it in the fixture means tests cannot reach it. Registration is mandatory. --- ## 8. Steps Module (Layer 3.5) When three or more ATCs run in the same order across three or more tests, the chain becomes a Step. Steps are NOT ATCs: no `@atc` decorator, not reported individually to the TMS, purpose is eliminating repetition in **preconditions**. The real pattern lives in `tests/components/steps/ExampleSteps.ts`: a Steps class extends `TestContext`, takes `TestContextOptions` in its constructor, and drives `this._page` / `this._request` directly. No fixture registration. ```typescript // tests/components/steps/AuthSteps.ts import type { TestContextOptions } from '@TestContext'; import { TestContext } from '@TestContext'; export class AuthSteps extends TestContext { constructor(options: TestContextOptions = {}) { super(options); } async authenticateUser(email: string, password: string): Promise<{ token: string }> { if (!this._request) { throw new Error('Request context not set. Pass { request } in constructor options.'); } const response = await this._request.post('/api/auth/login', { data: { email, password }, }); const body = await response.json(); return { token: body.token }; } // Object param: 3+ arguments never go positional (see typescript-patterns.md // §1). Three bare strings also read identically at the call site, so // swapping the last two is a silent bug. async navigateAsAuthenticatedUser(args: { path: string, email: string, password: string }) { const { path, email, password } = args; if (!this._page || !this._request) { throw new Error('Page and Request context must be set.'); } const auth = await this.authenticateUser(email, password); await this._page.evaluate(token => localStorage.setItem('authToken', token), auth.token); await this._page.goto(path); return auth; } } ``` Usage — instantiate the Steps class directly in the test (there is no `{ steps }` fixture): ```typescript import { AuthSteps } from '@steps/AuthSteps'; test('TICKET-ID: should display confirmation after checkout', async ({ ui, page, request }) => { const steps = new AuthSteps({ page, request }); await steps.navigateAsAuthenticatedUser({ path: '/checkout', email: config.testUser.email, password: config.testUser.password }); await ui.checkout.completeCheckoutSuccessfully(); await expect(page.locator('[data-testid="confirmation"]')).toBeVisible(); }); ``` When to use Steps vs direct ATC calls: | Scenario | Steps | Direct ATCs | |----------|-------|-------------| | Same 3+ ATC chain in 3+ test files | Yes | No | | One-off precondition for one test | No | Yes | | Complex setup with 5+ ATCs | Yes | No | | Simple single-ATC setup | No | Yes | --- ## 9. Tests Validate FLOWS Tests orchestrate ATCs; they do not implement logic. One test validates a complete flow with multiple assertions. ```typescript // WRONG — six tests, same call, different field checks test('should return orders', async ({ api }) => { ... }); test('should have referenceNumber', async ({ api }) => { ... }); // RIGHT — one test, complete contract test('TICKET-ID: should create order with correct totals when discount is applied', async ({ api }) => { const [, order] = await api.orders.createOrderSuccessfully(orderData); const totals = await api.orders.getTotals({ orderId: order.id }); expect(order.id).toBeDefined(); expect(order.discountApplied).toBe(true); expect(totals.finalAmount).toBe(totals.baseAmount - totals.discountAmount); }); ``` Separate tests only when the scenario is fundamentally different (different precondition, role, or outcome). Every `test()` must have the ticket ID as a prefix: `test('TICKET-ID: should {behavior} when {condition}', ...)`. One ticket per `describe`; never split a ticket across files. Multiple tickets can coexist in one file when they test different aspects of the same feature. Test independence: no shared state, each test generates its own data via `TestContext` helpers or faker, never rely on test ordering. --- ## 10. Decorators — @atc and @step ```typescript @atc('TICKET-ID') async signInSuccessfully(payload: SignInPayload) { ... } @atc('TICKET-ID', { softFail: true, severity: 'critical' }) async verifyOptionalField() { ... } @step async getCurrentUser() { ... } ``` | Aspect | `@atc` | `@step` | |--------|--------|---------| | Purpose | TMS traceability + tracing | Tracing only | | Apply to | Layer 3 state-changing ATCs | Layer 3 read-only helpers | | NDJSON export | Yes | No | `@atc` options: `softFail` (failure logs but does not block), `severity` for reporting. `severity` takes ALLURE's vocabulary — `'blocker' | 'critical' | 'normal' | 'minor' | 'trivial'` (`tests/utils/decorators.ts`) — because the value is forwarded straight to Allure. It is NOT the `critical/high/medium/low` scale used by the review checklists; passing one of those is a type error. Both decorators mask sensitive parameters (`password`, `token`, `secret`) in trace output. Never apply decorators to Layer 2 base methods or private helpers. Detailed tracing mechanics live in a separate tracing reference. `bun run kata:manifest` extracts every component and ATC into `kata-manifest.json`. **MUST be loaded before proposing any new Component or ATC** — Critical Rule #12 in `AGENTS.md`. The manifest is authoritative; the file system is not. Husky enforces freshness on commit (`bun run kata:manifest:check`), so the committed manifest is always trustworthy as the registry of record. -
planning-playbook.md 35 KB
# Planning Playbook — spec.md, automation-plan.md, atc/*.md Load during Phase 1 (Plan) of the Plan → Code → Review pipeline. Covers the three plan documents KATA automation uses, how to populate each by scope (Module / Ticket / Regression), the Discover → Modify → Generate data-classification workflow used while planning, and the approval gate between Plan and Code. Scope-selection rules (which scope to pick, the one-line summary of each) live in SKILL.md §"Pick the planning scope first". This file assumes the scope has been chosen and documents what to produce. > **Two plans, do not confuse them.** This playbook authors the **automation plan** (`automation-plan.md`) — a NON-Jira, hand-authored file living in the Epic's `test-specs/<scope>/` tree (committed to git). It is NOT the Story's dev `implementation-plan.md`, which is a Jira-synced, read-only per-field cache in the Story folder (`.context/PBI/epics/EPIC-<KEY>-<slug>/stories/STORY-<KEY>-<slug>/implementation-plan.md`) — read that as input via `bun run jira:sync-issues get <STORY-KEY>`, never hand-write it. The automation plan was historically named `implementation-plan.md`; it is renamed to `automation-plan.md` to avoid colliding with the Jira-synced dev plan. > **Path model.** All `test-specs/` artifacts live at the **Epic** level: `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/` (sibling of `stories/`). Module = Epic (1:1). `<scope>` = the ticket/regression slug or module slug. --- ## Plan dispatch (Single subagent) The Plan phase is delegated to a single subagent. The orchestrator does NOT read the KATA references, the existing component code, or the OpenAPI schemas during planning — that exploration lives entirely in the subagent's context. **Briefing** (7 components per `agentic-qa-core/references/briefing-template.md`): ``` Goal: Produce spec.md + automation-plan.md for scope <SCOPE> (module|ticket|ATC) <SCOPE_KEY>. Context docs: - kata-manifest.json (root) — REQUIRED FIRST READ. Authoritative registry of every existing Component + ATC. Use it for reuse detection and ID-collision avoidance before drafting anything. - .context/PBI/epics/EPIC-<KEY>-<slug>/stories/STORY-<KEY>-<slug>/ (Jira-synced caches: story.md, acceptance-criteria.md, implementation-plan.md (dev plan), acceptance-test-plan.md — READ-ONLY input; materialize via `bun run jira:sync-issues get <STORY-KEY> --include-comments`) - .context/master-test-plan.md - .context/business/business-data-map.md - .context/business/business-feature-map.md - .agents/skills/test-automation/references/kata-architecture.md - .agents/skills/test-automation/references/atc-tracing.md - tests/components/<api|ui>/ (existing components — open ONLY when the manifest entry is ambiguous) - api/schemas/ (TypeScript types for API tests) Project Standards (auto-resolved): <compact rules pulled from .agents/skills/REGISTRY.md per agentic-qa-core/references/skill-resolver.md — authoritative for listed conventions; do not re-read full SKILL.md files> Skills to load: (none — planning skill is loaded by orchestrator already) Exact instructions: 1. Load kata-manifest.json FIRST. Cross-check every candidate Component name against components.api[].name + components.ui[].name; cross-check every candidate ATC ID against components.{api,ui}[].atcs[].id. Treat any match as a reuse signal — never plan a duplicate. 2. Read remaining context docs to understand scope, business risks, and any coverage the manifest does not surface. 3. Draft spec.md with: scope summary, the TMS-ID table of TCs in scope, and the Automation Plan (order, shared fixtures, blockers). Do NOT restate TC bodies — they live in Jira and sync into test-cases/. 4. Draft automation-plan.md with: target file paths, fixture selection (api / ui / test), reused-vs-new components (cite manifest entries), dependency order, estimated complexity per ATC. 5. Write both files to .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope-slug>/. Report format: JSON: { "spec_path": "...", "plan_path": "...", "atc_count": <int>, "new_components": [...], "reused_components": [...], "open_questions": [...] } Rules: - Apply the inline-locator rule (locators inline in ATCs, extract only when reused 2+ times). - Apply ATC-identity rule (same output = one parameterized ATC). - Do NOT write actual test code — only spec + plan. - Surface open questions to the orchestrator instead of guessing. ``` The orchestrator reads the JSON report, surfaces open_questions to the user if any, and only proceeds to Code dispatch after user approval. --- ## 1. Plan document map Three document types, each tied to a scope. Every scope produces at least `spec.md`; ticket and regression scopes add `automation-plan.md`; complex ATCs add per-ATC specs under `atc/`. All live at the Epic level under `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/`. `test-specs/` is the one `[COMMIT]` island inside an otherwise gitignored Jira cache (`AGENTS.md` §9). These files describe the **test code**: they must land in the same commit as the code they produce, or a reviewer cannot contrast plan against implementation. That is also the line that decides what belongs here — a Jira `Test` issue holds the test case, an `atc/*.md` holds how to implement it in KATA. Same ID, two documents, two owners. | Document | Scope that produces it | Location | |----------|-----------------------|----------| | `spec.md` | Module (N specs), Ticket (1), Regression (1) | `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/{PREFIX}-T{NN}-{name}/spec.md` | | `automation-plan.md` | Ticket, Regression | Same folder as spec.md — `automation-plan.md` | | `atc/{TICKET-ID}-{brief-title}.md` | Complex ATCs from any scope | Same folder — `atc/{TICKET-ID}-{brief-title}.md` | | `ROADMAP.md`, `PROGRESS.md` | Module only | `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/` | Canonical folder naming: - `{module}` — kebab-case business area (`orders-dashboard`, `user-management`). - `{PREFIX}` — 2–3 letter module abbreviation (`OD`, `UM`, `AUTH`). Used for filesystem only — not for TC IDs. - `T{NN}` — zero-padded ticket slot (`T01`, `T02`, …). - `{name}` — ≤5-word kebab-case title. - `{TICKET-ID}` — TMS ticket key (e.g., `UPEX-123`). TC IDs in spec content must always be TMS-generated; never invent local IDs. --- ## 2. Inputs and outputs by scope ### 2.1 Module-driven (macro) Inputs: - Module name or feature area (`"Orders Dashboard"`, `"Billing"`). - Any stakeholder input: meeting transcript, priority list, known regressions. - Access to frontend and backend source for the module. - Access to `.context/` docs (business-data-map, api-architecture, existing PBI). Outputs: ``` .context/PBI/epics/EPIC-<KEY>-<slug>/ {module}-test-plan.md # Master document (Section 4) test-specs/ ROADMAP.md # Ticket index, phases, dependency graph PROGRESS.md # Session-persistent tracker {PREFIX}-T01-{name}/spec.md # 3–7 TCs per ticket {PREFIX}-T02-{name}/spec.md … ``` Scope targets: 3–7 TCs per ticket. A ticket with 10+ TCs is too broad — split it. Group tickets by functional area, not by UI section. ### 2.2 Ticket-driven (medium) Inputs: - Ticket ID in the TMS (e.g., `UPEX-101`). - Test type — `integration` or `e2e` (ask if not obvious). - Existing module context, if the ticket belongs to a module with prior `test-specs/`. Outputs: ``` .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/{PREFIX}-T{NN}-{name}/ spec.md # 1–7 TCs derived from the ticket's ACs automation-plan.md # Architecture decisions, ATC registry, scenarios, implementation order atc/*.md # Only for ATCs complex enough to warrant a per-ATC spec ``` Expected TC count: 1–7 per ticket (story-driven). ### 2.3 Regression-driven (micro) Inputs: - A single TC (often added after a bug fix) or a focused coverage gap. - Target component (existing or new). - Module name for folder placement. Outputs: ``` .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/{PREFIX}-T{NN}-{name}/ spec.md # 1–3 TCs (bug-driven) or 1–2 (gap-driven) automation-plan.md # Optional — skip when the spec is trivially one method atc/{TICKET-ID}-{brief-title}.md # Usually present — regression = one ATC in detail ``` A regression-driven plan is the smallest unit of work. Often it is just `spec.md` + `atc/*.md`, no automation plan, because one ATC is the whole body of work. --- ## 3. TMS-first principle (applies to every scope) TCs in `spec.md` must reference TMS-generated IDs, never local-only IDs. Before writing TCs: > **Prerequisite**: Load `/xray-cli` skill (Modality jira-xray) or `/acli` (Modality jira-native) before executing the TMS commands below. 1. Query the TMS for tests already linked to the ticket (via `[TMS_TOOL] List Tests` — resolve per AGENTS.md Tool Resolution). 2. **If TCs exist** — consume them as the base for `spec.md`; do not duplicate. 3. **If TCs are missing** — create them in the TMS first (`[TMS_TOOL] Create Test`), titled to the canonical form `{US_ID}: TC#: should <expected outcome> [<connector> <condition>] [given <precondition>]` (`#` = a stable per-Story index, never renumbered), capture the returned IDs, then write `spec.md`. 4. **If partial** — consume what exists, create the gaps in TMS under the same title form, write `spec.md` with the combined set. > A Test created here enters the regression repository the same way a promoted one does, so the same title rule binds — full grammar, the re-derive-then-verify order, and the anti-patterns: `test-documentation/SKILL.md` §"Naming — the one rule that matters" + §"Title on promotion". Local `{PREFIX}-T{NN}` naming is filesystem scaffolding. All TC headings inside `spec.md` use the TMS IDs (`### PROJ-101: should ...`). The same IDs become `@atc('PROJ-101')` decorators during the Code phase. --- ## 4. `spec.md` structure The spec is the **automation batch plan**: which TCs this scope automates, in what order, and what they share. It does NOT restate the test cases. > **Why it stopped carrying the Gherkin.** The TC body — preconditions, action, expected output, Gherkin — lives in the Jira `Test` issue, and `bun run jira:sync-issues` now materializes every Test linked to a Story into `test-cases/TEST-<KEY>-<slug>.md` under that Story. Copying it here too put the same text on disk twice, and the copy nobody re-synced was the one people read. Reference the TMS ID; the body is one sync away. > **Reference TCs by Jira key, never by path.** Folder slugs are derived from issue summaries, so a Story retitled in Jira renames its folder and breaks every hardcoded link — relative or aliased. The key is the only stable identifier. ```markdown # {PREFIX}-T{NN}: {Title} | Field | Value | |-------|-------| | **Priority** | P0 / P1 / P2 | | **Phase** | {phase number + name, or "Standalone"} | | **Items** | {N} TCs {+ M multi-ATC tests if any} | | **Dependencies** | {other ticket IDs, or "None"} | | **Requires** | {test data, accounts, env conditions} | | **Source** | Story: {TICKET-ID} / Bug: {brief} / Gap: {brief} | ## Summary {What this scope automates and why it matters. 2–4 sentences.} ## Test Cases > Bodies live in Jira. Synced copies sit under the covering Story's `test-cases/` > (or `epics/_orphans/tests/` when the Test covers nothing); run > `bun run jira:sync-issues get {TICKET-ID}` if they are not on disk. | TMS ID | Title | Type | Priority | |--------|-------|------|----------| | {TMS_TC_ID} | should {behavior} when {condition} | Positive / Negative / Boundary | P0 | ## Automation Plan **Order**: {execution order and why — e.g. "PROJ-501 first: it seeds the coupon the rest reuse"} **Shared fixtures**: {fixtures these TCs have in common, and which ATC creates each} **Blocked by**: {missing mock, unavailable env, unbuilt helper — or "Nothing"} **Preconditions for the whole scope**: {entities, states, accounts required before any of these run — distinct from a single TC's own preconditions, which live in Jira} ## Merged TCs (if any) | Removed ID | Merged Into | Reason | ## Updated TCs (if any) | TC ID | Spec File | What Was Added | Reason | ## Acceptance Criteria - [ ] {N} TCs automated with {pattern description} - [ ] Tests pass on local and staging ``` Rules for `spec.md`: - Every row in the Test Cases table carries a **TMS-generated ID**. A local-only ID means the TC was never created in the TMS — go create it first (§3). - Do NOT paste the Gherkin, the steps, or the expected output. If you find yourself explaining what a TC verifies, that belongs in the Jira issue. - Multi-step flows (2+ actions with intermediate verifications) are flagged as multi-ATC tests, not a single TC. - Priority levels: P0 (release-blocker), P1 (high value), P2 (edge cases). - The Automation Plan section is the part only this file can hold: order, shared fixtures and blockers are properties of the batch, not of any one test case. --- ## 5. Module-scope master document (`{module}-test-plan.md`) Module scope alone produces this master doc in addition to the per-ticket specs. It is the single source of truth for module-level context that every ticket spec will reference. Sections, in order: 1. **Executive Summary** — why the module matters, key risks, stakeholder priorities. 2. **Module Overview** — what it is, who uses it, how it connects to other modules. 3. **Page/API Architecture** — visual states, panel layout, conditional rendering, status flows. 4. **Data Flow & API Endpoints** — all endpoints the page calls, calculation formulas, refresh triggers. 5. **Test Scenarios (Gherkin)** — organised by functional area, each scenario with Preconditions → Action → Expected. 6. **Implementation Roadmap** — phases ordered P0 → P2 with dependencies between tickets. 7. **Test Data Strategy** — per-ticket feasibility table (Discover / Modify / Generate — see §7). 8. **Key Selectors Reference** — CSS selectors or `data-testid` values the automation will target. Write it as if the reader has zero context. Include calculation formulas explicitly, stakeholder quotes where they describe priorities, and ASCII diagrams for layouts. Diagrams beat paragraphs. --- ## 6. `automation-plan.md` structure The automation plan is the technical contract — what code to write, which components already exist, which ATCs to reuse vs create, the implementation order. (Hand-authored in `test-specs/`; distinct from the Story's Jira-synced dev `implementation-plan.md`.) ```markdown # Test Automation Plan: {TICKET-ID} > Ticket: {TICKET-ID} — {Title} > Type: integration | e2e > Sprint: {Sprint Name} > Created: {date} ## 1. Ticket Summary - What to test - Acceptance Criteria (list) - Dependencies (other ticket IDs) ## 2. Architecture Decisions > **ADR promotion check.** The decisions in this section are **ticket-local** by default — they stay in this plan. If one is **architectural AND hard to reverse** (a fixture lifecycle reused across tickets, a test-data-isolation contract, a Page-Object-vs-Screenplay shift, an auth-in-tests change, a flake-retry policy), it is no longer ticket-local: promote it to a standalone `ADR-NNNN-<slug>.md` in `.context/ADR/` and leave a `See ADR-NNNN` backlink in the table below. Detection + procedure: `agentic-qa-core/references/adr-doctrine.md` §1–§2; template + lifecycle: `.context/ADR/README.md`. AI drafts `Proposed`; the human approves. ### Component Strategy | Decision | Value | Rationale | | Component | {Resource}Api.ts / {Page}Page.ts | New or existing? | | Fixture | { api } / { ui } / { test } | Why this fixture? | | Test file | tests/{type}/{module}/{verbFeature}.test.ts | Naming rationale | | Preconditions | Steps module / inline | What setup is needed? | ### [E2E ONLY] UI Elements | Element | Locator Strategy | Locator Value | ### [INTEGRATION ONLY] API Details | Aspect | Value | | Endpoint(s) | METHOD /api/v1/... | | OpenAPI Type(s) | {TypeName} from @schemas/... | | Auth Required | Yes / No | | Return Pattern | Tuple: [APIResponse, TBody] or [APIResponse, TBody, TPayload] | ## 3. ATC Registry ### Existing ATCs (Reuse) | ATC ID | Component | Method | Description | ### New ATCs (Create) | ATC ID | Component | Method | Description | ### New Helpers (No @atc) | Component | Method | Returns | Description | ## 4. Test Data Strategy | Data | Source | Lifecycle | ### DataFactory additions / Constants additions (code snippets) ## 5. Test Scenarios ### File: tests/{type}/{module}/{verbFeature}.test.ts Fixture: { api } / { ui } / { test } #### Scenario 1: {happy path} Test: "{TICKET-ID}: should {behavior} when {condition}" Preconditions: ... ATCs called: ... Test-level assertions: ... Teardown: ... ## 6. Implementation Order - [ ] Add types to tests/data/types.ts - [ ] Add factory methods to tests/data/DataFactory.ts - [ ] Add constants to tests/data/constants.ts - [ ] Create/update Layer 3 component - [ ] Register component in fixture - [ ] Create test file - [ ] Run tests and validate - [ ] TMS TC transitions to Pull Request when the PR opens (`create_pr` transition) — Automated only lands post-merge via the `merged` transition ## 7. Success Criteria - [ ] All ACs covered — **the floor, not the bar.** Also: risk-beyond-AC covered (invalid/boundary inputs, auth/error paths, state transitions, anomalies the AC is silent on) per `agentic-qa-core/references/test-design-doctrine.md` - [ ] Boundary cases (BVA) automated wherever an AC has a range / limit / length / date-window (EP-merge does NOT cover off-by-one) - [ ] KATA compliance - [ ] Fixture correct - [ ] No hardcoded waits - [ ] Aliases used - [ ] Tests pass locally - [ ] TMS TC moved to Pull Request on PR open (never "Automated" after local validation — Automated is exclusively the post-merge `merged` transition, once CI is green on main) ``` Implementation-order rule of thumb: each box in Section 6 should map to a single commit. If a commit would mix "new types" and "new ATC", split it. --- ## 7. Data classification — Discover / Modify / Generate Every precondition in a plan must be classified by how its data will be obtained. Apply this during planning — never defer to the coding phase. Priority is strict: try Discover first, then Modify, then Generate. | Priority | Pattern | Description | Feasibility check | |----------|---------|-------------|-------------------| | 1 | **Discover** | Query the system for existing data already in the required state. Zero DB impact. Preferred. | Can DB/API return an entity matching the preconditions? | | 2 | **Modify** | Find existing data and alter it via API to reach the required state. | Does the API expose the mutation needed? | | 3 | **Generate** | Create data from scratch via API (faker payload + POST). Last resort. | Does a POST/PUT endpoint exist for this entity? | | — | **Blocker** | No pattern is feasible. | Flag ticket as NOT automatable; document the gap and escalate. | Workflow during planning: 1. List each precondition (`user in X role`, `order in Y status`, `feature flag Z enabled`). 2. For each one, pick the lowest-priority pattern that works. 3. Document the chosen pattern, the exact endpoint or DB query that obtains the data, and where it runs (`beforeAll`, `beforeEach`, test-level setup). 4. For Modify and Generate, document cleanup/teardown (a test that creates state owns its cleanup). Rules: - Never hardcode entity IDs, usernames, or dates. Fetch dynamically at runtime or generate via faker. - Discover queries run in `beforeAll`. If the query finds nothing, the test uses `test.skip()` — never `expect(...).toBe(...)` on precondition data. Unrelated tests must not be blocked by missing data for one test. - If two tests need different states of the same entity, each test creates its own record. No shared mutable state. - Auth tokens and credentials always come from `.env` via the project's variables module — never generated and never hardcoded. Record a feasibility row per ticket inside the plan (module master doc §7 and automation-plan §4): ``` | Ticket | Precondition | Pattern | Feasibility | Notes | | OD-T01 | user with 0 orders | Discover | Risky | Query may be slow — add timeout guard | | OD-T02 | order with discount applied | Generate | Feasible | POST /orders then POST /orders/{id}/discount | ``` --- ## 8. `atc/{TICKET-ID}-{brief-title}.md` structure Produce a per-ATC spec when the ATC is complex enough that the implementation plan cannot carry the detail, or when a regression-driven plan is just one ATC. Template (abbreviated — full sections below): ```markdown # ATC Spec: {TICKET-ID} — {ATC Name} > Ticket: {TICKET-ID} > Component: {ComponentName} (tests/components/{api|ui}/{ComponentName}.ts) > Type: API | UI — Mutation | Verification | Negative | Happy path | Validation | Navigation | State change > Parent Story: {PARENT-TICKET-ID} (if applicable) ## 1. Test Case Summary | Name | Objective | Precondition | Acceptance Criteria | ## 2. ATC Contract \`\`\`typescript /** * ATC: brief description * Fixed assertions: * - ... */ @atc('{TICKET-ID}') async {methodName}({params}): {ReturnType} { /* ... */ } \`\`\` ## 3A. API Details (API ATCs) — endpoint, return type, OpenAPI imports, request/response shapes ## 3B. UI Details (UI ATCs) — page path, locator strategy, Playwright assertions, intercept patterns ## 4. Assertions Split ### Fixed (inside ATC) — invariants that always hold ### Test-level (in test file) — varies per scenario ## 5. Code Template — copy-pasteable skeleton with placeholders ## 6. Technique-derivation check (decides the ATC set — 1:N per AC) > Full canon + triggers: `agentic-qa-core/references/test-design-doctrine.md`. EP-merge collapses inputs only WITHIN a partition — never across partitions, boundaries, or states. | AC | Technique fired | ATCs produced | |----|-----------------|---------------| | (per AC) | EP (always) / BVA (range·limit·length·date) / State-Transition (status field) / Decision Table (2+ interacting conditions) / Pairwise (3+ factors) | … | **Equivalence Partitioning detail:** | Input | Expected output | Same ATC? | **Boundary Value Analysis detail** (mandatory if any range/limit exists — else state N/A): | Field + range | Boundary ATCs (`min-1·min·min+1 … max-1·max·max+1`, zero/empty/null) | **Two reduction axes — keep them separate** (canon: doctrine §"Part 2.5"): - **EP/BVA decide the ATC COUNT** — how many parameterized `@atc`s the cases split across (one per partition + boundary + state). - **Decision Table / Pairwise reduce the DATA ROWS inside one parameterized ATC** — when an ATC's data set combines 2+ interacting conditions (Decision Table → one row per surviving rule) or 3+ factors (Pairwise → all-pairs rows instead of the full cartesian product). They shrink the fixture/`data-factory` row set, NOT the ATC count. Log the reduction in the fixture or the spec so it is visible, not a silent cap. | Parameterized ATC | Reduction applied | Rows after reduction | |---|---|---| | (per multi-factor ATC) | Decision Table / Pairwise / none | … | ## 7. Dependencies - Precondition Steps - Required Components (exists? action needed) ## 8. Data Context (skip if parent plan covers it) | Precondition | Pattern | Source | Placement | Cleanup | ## 9. Checklist - [ ] verb{Resource}{Scenario} naming - [ ] Max 2 positional params - [ ] Correct return type (tuple for API, void for UI) - [ ] Fixed vs test-level assertions split - [ ] Not duplicating an existing ATC (EP checked) ``` ATC classification during planning: | Type | API trigger | UI trigger | Fixed assertion shape | |------|-------------|------------|----------------------| | Mutation | POST/PUT/PATCH/DELETE | Fill + Submit + Navigate | Status 2xx + created/updated fields | | Verification | GET + business-rule check | State-change verification | Status + business-rule invariants | | Negative | Any (expects 4xx/5xx) | Invalid submit | Error status + error contract | | Validation (UI) | — | Invalid form submit | Error visible, no navigation | | Navigation (UI) | — | Click + destination | URL + heading + key elements | Disguised helpers — if the method only does a GET with a status-200 assertion, or only a click without outcome assertions, it is a helper (no `@atc`), not an ATC. --- ## 9. Using `kata-manifest.json` during planning `kata-manifest.json` (root) is the authoritative registry of every component and every `@atc('ID')` call in `tests/components/**`. **MUST be loaded before drafting any plan** — Critical Rule #12 in `AGENTS.md`. The husky pre-commit gate keeps the file fresh, so the manifest is always trustworthy; the file system is not (a freshly added component may exist on disk but the manifest is what reviewers and downstream agents consult). Regenerate when stale: `bun run kata:manifest`. Validate: `bun run kata:manifest:check` (CI-grade; exits 1 if stale). Planning tasks the manifest answers: | Need | How | |------|-----| | "Does an `OrdersApi` component already exist?" | Look under `components.api[].name` | | "Is ATC `UPEX-101` already decorated somewhere?" | Grep `atcs[].id` in every component | | "Which component owns endpoint X?" | Component names map to domain; confirm by opening the file only if unclear | | "Which Steps classes already compose ATCs?" | Check the manifest's steps listing (`tests/components/steps/` scan) and its method list | Include two tables in the implementation plan based on manifest output: - **Existing ATCs (Reuse)** — populated from manifest entries whose `id` matches ACs already covered. - **New ATCs (Create)** — ATC IDs that must be created for this ticket. A planned "new component" that the manifest already lists is a duplicate and will be rejected in review. A planned `@atc('PROJ-XXX')` ID that already appears in `atcs[].id` is an ID collision and will be rejected. Both errors are avoidable by reading `kata-manifest.json` first. --- ## 10. Approval gate — Plan → Code Never start Phase 2 (Code) without a written plan the user has approved. The gate is structural, not procedural politeness: it prevents the most common failure mode (coding the wrong scope). Gate checklist: - [ ] `spec.md` exists and every TC has a TMS-generated ID. - [ ] For ticket/regression scope: `automation-plan.md` exists with §3 ATC Registry populated. - [ ] For complex ATCs: `atc/*.md` exists with the contract (signature + fixed assertions) defined. - [ ] Data strategy is documented per precondition (pattern + source + placement + cleanup). - [ ] Fixture decision is recorded (`{ api }` / `{ ui }` / `{ test }`). - [ ] Every "New ATC" in the registry has a unique `@atc` ID that does not collide with `bun run kata:manifest` output. - [ ] Module master doc exists (module scope only) with §4 Data Flow and §7 Data Strategy populated. - [ ] Implementation order is defined with one commit per step. Presentation to the user: 1. Summarise the scope (module / ticket / regression). 2. List the TCs from `spec.md` (IDs + titles). 3. List the ATCs to be created and reused from `automation-plan.md` §3. 4. Flag any preconditions classified as Risky or Blocker in §7 — the user decides whether to proceed, adjust scope, or defer. 5. Wait for explicit approval before moving to Phase 2. On approval, Phase 2 begins. If approval is not forthcoming, revise the plan — do not start coding with an unapproved plan. On rejection, document the reason in the plan and iterate. --- ## 11. Checklists by scope ### Module-driven - [ ] Parallel context gathering complete (frontend, backend, existing `.context/` docs). - [ ] Master document written with all 8 sections (§5). - [ ] Tickets grouped by functional area (not by page section). - [ ] Each ticket has 3–7 TCs; none exceed 10. - [ ] Equivalence Partitioning applied across TCs; merges documented. - [ ] Every TC has a TMS-generated ID (§3). - [ ] ROADMAP.md, PROGRESS.md created. - [ ] Data strategy table filled for every ticket (§7). ### Ticket-driven - [ ] Story materialized via `bun run jira:sync-issues get <STORY-KEY> --include-comments`; ACs + dev implementation-plan + ATP read from the synced `.md` (NEVER `acli workitem view` for custom fields). - [ ] TMS queried for existing tests; missing ones created there first. - [ ] `spec.md` written with 1–7 TCs. - [ ] `automation-plan.md` §3 ATC Registry populated against `kata:manifest` output. - [ ] Fixture decision made and justified. - [ ] Every precondition classified (§7). - [ ] Implementation order written with one commit per step. ### Regression-driven - [ ] Bug or gap is well-understood; the TC that would have caught it is written. - [ ] TMS TC exists (1–3 bug, 1–2 gap). - [ ] ATC spec written (`atc/*.md`) because the regression is one method's worth of work. - [ ] Data strategy documented on the ATC spec (§8 in the ATC template) since there is often no parent automation-plan. - [ ] Component placement decided — existing component vs new + fixture update. When every box is checked, the plan is ready to hand off to Phase 2 (Code). Until then, the plan is not complete and the approval gate (§10) has not been reached. --- ## 12. Interrupted-session recovery When `/test-automation` is invoked mid-flow (or resumed after context loss), the FIRST stop is the mandatory Phase 0 session contract: read `.session/test-automation/<scope>/{plan.md, progress.md}` per `agentic-qa-core/references/session-management.md` and offer resume / restart / abort. `PROGRESS.md` + `ROADMAP.md` (in `test-specs/`) are the module-batch trackers — read them to confirm which ticket the resume applies to, but they complement the `.session/` contract, they do not replace it. | Has plan? (`automation-plan.md`) | Has test code? (`tests/e2e/**` or `tests/integration/**`) | Resume from | |---|---|---| | No | No | **STEP 2 (Planning)** | | Yes | No | **STEP 3 (Coding)** | | Yes | Yes | **STEP 4 (Review)** | Before classifying state, read the current ticket in `PROGRESS.md` §Current status to confirm which ticket the resume applies to. ### 12.1 Revision-loop ceiling (Phase 3 · Review) Maximum revision loops: **2**. If the test is still not APPROVED after 2 revision rounds, present all remaining issues to the user and ask for guidance. Do not enter an infinite loop of reviewer ↔ coder mutations. ### 12.2 Quality Gates G1–G4 Named phase-transition checkpoints. Each gate blocks progression until its criteria are met. | Gate | Between | Criteria | |---|---|---| | **G1 · Plan exists** | STEP 2 → STEP 3 | `automation-plan.md` created with ATCs defined (see §10 Approval gate) | | **G2 · Tests pass** | STEP 3 → STEP 4 | All ATCs green locally — soft override allowed only after §12.3 bug-detection sub-protocol | | **G3 · Review OK** | STEP 4 → STEP 5 | Reviewer verdict = APPROVED, or §12.1 ceiling (2 rounds) reached and the user decided next steps | | **G4 · Progress updated** | STEP 5 → STEP 6 | `PROGRESS.md` reflects the completed ticket (status, test file path, done count, Session Log entry) | ### 12.3 G2 failure protocol — legitimate bugs during automation If G2 fails because a test uncovers a real product bug (not flaky, not a coding error), follow this sub-protocol **before** invoking the soft override: > **Prerequisite**: Load `/acli` skill before executing the `[ISSUE_TRACKER_TOOL]` commands below. 1. **Search the issue tracker** for an existing bug that matches the observed vs expected behaviour (`[ISSUE_TRACKER_TOOL] Search Issues`). 2. **If the bug is already reported**: - Add a test annotation tying the failure to the bug key — e.g. `test.fail('Blocked by {BUG-KEY}')` or a `@blocked:{BUG-KEY}` tag. - Keep the test in the suite during automation runs (no silent skip — the failure is informative). - The `@blocked:{BUG-KEY}` tag lets `/regression-testing` filter the test out of GO/NO-GO decisions until the bug is fixed. 3. **If the bug is NOT reported**: - Present the observed vs expected diff to the user and wait for explicit confirmation that it is a real defect. - **Classify it first** — Bug vs Defect vs Improvement by the FEATURE's lifecycle stage, NOT by where the failing test ran (feature live above Staging → Bug; still pre-release → Defect; under-specified/absent AC surfaced by a test-beyond-AC → Improvement), per `agentic-qa-core/references/defect-management-doctrine.md` (Part 1). Also identify the affected product component. (This is the Jira classification — independent of the `@atc(...)` Allure severity, which is reporting metadata, not the Jira severity field.) - On approval, delegate report creation to `/sprint-testing` (Stage 3 Reporting — see `sprint-testing/references/reporting-templates.md` §1), **passing the classification + affected component in the handoff** so `/sprint-testing` files the correct issue type. `/sprint-testing` performs the actual filing (QA Assignee, components, QA process epic). - Once the issue key is issued, apply step 2 above. 4. **Document in `PROGRESS.md`** — record each blocked test + bug key in the Session Log (and the Blocked tests table, see §13.2) so the next session does not re-investigate the same failure. Only after steps 1–4 can G2 be overridden and STEP 4 (Review) start. A failing test without a bug key behind it is never an acceptable override. --- ## 13. Shared state files — templates Both files live under the Epic's `test-specs/` (`.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/`) and are the single source of truth for module-wide automation progress. Populate the template blocks verbatim (copy, then fill); keep section headings stable so future sessions can grep reliably. ### 13.1 `ROADMAP.md` (ticket index + dependencies) ````md # {MODULE} · Automation Roadmap ## Tickets | ID | Title | Priority | Phase | Dependencies | TCs | |---|---|---|---|---|---| | {PREFIX}-T01-{slug} | ... | P0 | Plan | — | 3 | | {PREFIX}-T02-{slug} | ... | P1 | Code | T01 | 5 | ## Dependency graph ``` T01 ──┬──► T02 ──► T04 └──► T03 ``` ## Phase progress - Plan: ▓▓▓▓░ 4/5 - Code: ▓▓░░░ 2/5 - Review: ▓░░░░ 1/5 ```` ### 13.2 `PROGRESS.md` (session-persistent tracker) ````md # {MODULE} · Automation Progress ## Current status - Current ticket: {PREFIX}-T02-{slug} - Completed: 1/5 - Remaining: 4/5 ## Tickets | Ticket | Status | Test file | Done | Notes | |---|---|---|---|---| | T01 | done | tests/e2e/login.test.ts | 3/3 | — | | T02 | in-progress | tests/e2e/signup.test.ts | 1/5 | Fixture blocked on email-verify mock | ## Session log | Date | Action | Actor | Artifacts | |---|---|---|---| | 2026-04-19 | Planned T02 | AI | automation-plan.md | | 2026-04-20 | Coded T02 ATC1 | AI | tests/e2e/signup.test.ts | ## Shared components created - `UserFormPage` (`tests/components/ui/UserFormPage.ts`) — used by T02, T05 - `AuthApi.signupWithRetry` — used by T02, T04 ## Blocked tests | Test | Bug key | Reason | Since | |---|---|---|---| | `signup > invalid email rejects` | UPEX-999 | Server-side validation missing | 2026-04-20 | ```` The Blocked tests table is populated from §12.3 — every test marked `test.fail('Blocked by {BUG-KEY}')` + tagged `@blocked:{BUG-KEY}` (the blocked-by-bug convention defined in `automation-standards.md` §7, distinct from `softFail` and `@flaky`) must appear here with the bug key, the reason, and the date the block began. Remove a row only when the bug is closed and the test goes green. -
review-checklists.md 22.1 KB
# Review Checklists — E2E and API Deltas Load during Phase 3 (Review) of the Plan → Code → Review pipeline. This file is a **delta**: it assumes the shared KATA review checklists from `automation-standards.md` §10 (Component review, ATC review, Test file review) have already been applied. Only the E2E-specific and API-specific additions live here, plus the final handoff checklist that gates "ready for CI". Severity model for all findings below matches the shared severity scale — CRITICAL (blocks merge), HIGH (blocks merge), MEDIUM (recommended), LOW (nice to have). --- ## Parallel verification dispatch (Phase 3 Review) `bun run test`, `bun run types:check`, and `bun run lint:check` are independent — they don't share state — so dispatch all three simultaneously and aggregate. **Three briefings**, each a Single Parallel subagent. Dispatch them in ONE tool-call block. ### Verifier A — test runner ``` Goal: Run the test suite and report failures grouped by file. Context docs: - playwright.config.ts (project list) - .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/spec.md (which ATCs should pass) Skills to load: (none) Exact instructions: 1. Run: bun run test (the script declared in package.json invokes `playwright test`). 2. Capture: total / passed / failed / skipped counts. 3. For each failure: file, test name, error message (first 200 chars), trace path if available. Report format: JSON: { "verifier": "test", "passed": <int>, "failed": <int>, "duration_seconds": <int>, "failures": [ { "file": "...", "test": "...", "error": "..." } ] } Rules: - Do NOT edit any test code. - Do NOT retry failed tests. ``` ### Verifier B — type-check ``` Goal: Run TypeScript compiler in noEmit mode and report errors. Context docs: - tsconfig.json Skills to load: (none) Exact instructions: 1. Run: bun run types:check (the script declared in package.json invokes `tsc --noEmit`). 2. Capture every error: file:line:col, error code, message. Report format: JSON: { "verifier": "type-check", "errors": [ { "file": "...", "line": ..., "col": ..., "code": "TS...", "message": "..." } ] } Rules: - Do NOT auto-fix. - Strict mode is mandatory; do not relax tsconfig. ``` ### Verifier C — lint ``` Goal: Run ESLint and report violations. Context docs: - eslint.config.* / .eslintrc.* Skills to load: (none) Exact instructions: 1. Run: bun run lint:check (the script declared in package.json invokes `eslint .`). 2. Capture every violation: file:line, rule, message, severity. Report format: JSON: { "verifier": "lint", "violations": [ { "file": "...", "line": ..., "rule": "...", "message": "...", "severity": "error|warning" } ] } Rules: - Do NOT pass --fix. - Treat warnings as informational, errors as blockers. ``` ### Aggregation in the main thread After all three Verifiers return, the orchestrator: 1. Counts total blockers (failed tests + type errors + lint errors). 2. Decides: 0 blockers → merge candidate; >0 → reject + surface report to user. 3. If reject: presents the consolidated list to the user; does NOT auto-fix. ### Fallback to serial If the test suite is large enough that running 3 verifiers in parallel saturates the machine (CPU/IO contention), fall back to serial. The dispatch overhead vs latency tradeoff flips below ~30s total runtime. --- ## 1. Before you review — apply the shared list first Run these **three shared checklists first**, in order. They cover the rules that are identical for E2E and API tests: file naming, class extension, decorator placement, ATC identity, tuple returns, locator rule, max-2-positional-param rule, inline assertions, fixture registration, no relative imports, ticket-ID prefix, no `test.only`/`test.skip` left behind, no hardcoded waits. 1. **Component review** — `automation-standards.md` §10 → "Component review" 2. **ATC review** — `automation-standards.md` §10 → "ATC review" 3. **Test file review** — `automation-standards.md` §10 → "Test file review" Only after the shared list is clean should you layer the deltas in §2 (E2E) or §3 (API) below. Do not merge a PR until both the shared checklist and the applicable delta are clean. ### Severity mapping at a glance | Level | Examples in the deltas below | |-------|-------------------------------| | CRITICAL | Missing `@atc` / wrong tuple shape / no status-code assertion (API) / hardcoded `waitForTimeout` in E2E | | HIGH | Brittle selector (nth-child, class-only) / missing type generic on `apiGET`/`apiPOST` / missing auth `beforeEach` on protected endpoint / no 401 coverage for protected routes | | MEDIUM | Missing JSDoc / test not tagged / helper disguised as ATC / hand-rolled interface duplicating OpenAPI type | | LOW | Import order / blank-line churn / variable naming | ### Output format Keep reviewer output consistent across E2E and API: ```markdown # Review: {TICKET-ID} ## Summary | Category | Pass | Fail | |----------|------|------| | Shared (automation-standards §10) | … | … | | Delta (this file §2 or §3) | … | … | ## Verdict - [ ] APPROVED - [ ] NEEDS REVISION (CRITICAL/HIGH present) - [ ] MINOR CHANGES (only MEDIUM/LOW) ## Findings (by severity) ... ``` --- ## 2. E2E-specific delta (UI / Playwright) Apply these checks in addition to the shared list for any file under `tests/e2e/**` or any `*Page.ts` component. ### 2.1 Locator quality | ID | Check | Severity | |----|-------|----------| | E-L1 | Locators prefer `data-testid` or `getByRole` before CSS / XPath. | HIGH | | E-L2 | No brittle selectors: no `nth-child(...)`, no class-only selectors like `.btn.primary`, no deep descendant chains. | HIGH | | E-L3 | Locators are specific enough that a parallel test cannot accidentally hit the same node. | MEDIUM | | E-L4 | Locators live inline in the ATC. A selector reused in 2+ ATCs is promoted to a `private readonly` arrow function on the class — not to a `locators/*.ts` file. | CRITICAL (rejection reason in `automation-standards.md`) | | E-L5 | Text selectors are wrapped so translations do not break the test (e.g. `getByRole('button', { name: /submit/i })`). | MEDIUM | ### 2.2 Waits and synchronisation | ID | Check | Severity | |----|-------|----------| | E-W1 | No `page.waitForTimeout(n)` anywhere. Arbitrary sleeps are a CRITICAL reject. | CRITICAL | | E-W2 | Waits are condition-based: `waitForSelector`, `waitForResponse`, `waitForLoadState('networkidle')`, or `expect(locator).toBeVisible()`. | CRITICAL | | E-W3 | `data-loaded="true"` attributes or similar readiness flags are used where available. | MEDIUM | | E-W4 | No reliance on Playwright retries to mask race conditions — `retries: 0` remains the local default. | HIGH | ### 2.3 Visual and content gotchas | ID | Check | Severity | |----|-------|----------| | E-V1 | Text assertions avoid exact strings that vary by locale, time, or currency format — use regex or structural assertions. | MEDIUM | | E-V2 | Toasts / notifications are awaited with `expect(toast).toBeVisible()` before assertion (they disappear). | HIGH | | E-V3 | Screenshot assertions (`toHaveScreenshot`) are gated behind a Playwright project to avoid running on every test run unless intended. | MEDIUM | | E-V4 | Viewport is set explicitly for tests that depend on responsive breakpoints. | LOW | ### 2.4 Session reuse and authentication | ID | Check | Severity | |----|-------|----------| | E-S1 | Logged-in tests consume a storage state from auth setup — they do not re-run the login ATC per test. | HIGH | | E-S2 | Each test that mutates user state uses a fresh user — no shared mutable logged-in session across tests. | HIGH | | E-S3 | Session cookies or local-storage tokens are cleared in `beforeEach` for 401 / logged-out tests. | HIGH | ### 2.5 iframe, shadow DOM, new tabs | ID | Check | Severity | |----|-------|----------| | E-F1 | iframe interactions use `page.frameLocator(...)`, not raw `page.$` on the outer document. | HIGH | | E-F2 | Shadow-DOM nodes are reached through `locator(':light(...)')` or explicit `shadowRoot` chains — never by brittle CSS that happens to pierce the boundary. | HIGH | | E-F3 | Popups / new tabs are awaited via `context.waitForEvent('page')`, not via fixed timeouts. | CRITICAL | | E-F4 | File downloads use `page.waitForEvent('download')`. Uploads use `setInputFiles`. | HIGH | ### 2.6 Screenshot, trace, video policy | ID | Check | Severity | |----|-------|----------| | E-T1 | Screenshots and traces are configured at the Playwright project level (`screenshot: 'only-on-failure'`, `trace: 'retain-on-failure'`) — not called manually inside ATCs. | MEDIUM | | E-T2 | The ATC does not log raw passwords or tokens. The `@atc` decorator masks parameters whose keys are in the sensitive set — verify parameter names are canonical (`password`, `token`, `secret`). | HIGH | | E-T3 | `page.screenshot()` inside a helper is deleted unless the screenshot is part of the test's explicit evidence contract. | MEDIUM | ### 2.7 Navigation | ID | Check | Severity | |----|-------|----------| | E-N1 | `goto()` is a plain method (undecorated or `@step`), not an `@atc`. Navigation is not a test case. | HIGH | | E-N2 | Absolute URLs are not hardcoded — use `this.buildUrl(path)` or the base URL from `@variables`. | HIGH | | E-N3 | Back-button / history navigation is asserted explicitly when behaviour depends on it. | LOW | --- ## 3. API-specific delta (Integration / HTTP) Apply these checks in addition to the shared list for any file under `tests/integration/**` or any `*Api.ts` component. ### 3.1 OpenAPI facade compliance | ID | Check | Severity | |----|-------|----------| | A-O1 | Response and payload types come from `@schemas/{domain}.types` — no direct `@openapi` imports in the component. | HIGH | | A-O2 | No hand-rolled interface duplicates a schema that already exists in the OpenAPI types file. If found, replace with the generated type. | MEDIUM | | A-O3 | New endpoints are accompanied by a `bun run api:sync` so the generated types reflect the contract. | HIGH | | A-O4 | `api/schemas/index.ts` re-exports any new domain added — otherwise `@schemas/{domain}` will fail to resolve. | HIGH | | A-O5 | Error responses use `ApiErrorResponse` (or the project's canonical error type), not the success shape. | HIGH | ### 3.2 HTTP method and tuple discipline | ID | Check | Severity | |----|-------|----------| | A-H1 | `apiGET<TBody>()`, `apiPOST<TBody, TPayload>()`, `apiPUT<TBody, TPayload>()`, `apiPATCH<TBody, TPayload>()`, `apiDELETE<TBody>()` — type generics always present. | HIGH | | A-H2 | Return tuples match the HTTP verb — see expanded table below (§3.2.1). | CRITICAL | | A-H3 | ATC `return` statement includes the payload on POST/PUT/PATCH — not just `[response, body]`. | CRITICAL | | A-H4 | `baseEndpoint` constant is defined once per component; individual calls compose paths from it. | MEDIUM | #### 3.2.1 Tuple-return contract per HTTP method | Method | Return shape | Notes | |---|---|---| | `GET` (single) | `[APIResponse, TBody]` | `TBody` = resource DTO | | `GET` (list) | `[APIResponse, TBody[]]` *or* `[APIResponse, ListResponse<TBody>]` | Use `ListResponse<T>` when the API wraps the collection with pagination metadata | | `POST` | `[APIResponse, TBody, TPayload]` | `TPayload` = the body that was sent | | `PUT` / `PATCH` | `[APIResponse, TBody, TPayload]` | Same shape as `POST` | | `DELETE` | `[APIResponse, TBody]` | Default: the API echoes the deleted resource as `TBody`. **If the target API responds `204 No Content`, switch to `[APIResponse, void]` and document the choice at the top of the owning `*Api.ts` component** (one-line JSDoc is enough). Whichever shape applies, every `DELETE` ATC in the component must use it consistently. | ### 3.3 Status-code and body assertions | ID | Check | Severity | |----|-------|----------| | A-A1 | Every ATC asserts the HTTP status code inline (`expect(response.status()).toBe(2xx/4xx/5xx)`). | CRITICAL | | A-A2 | Every success ATC asserts at least one body invariant (an ID is set, a field is present, a state is correct). | CRITICAL | | A-A3 | Inline assertions are invariants for the ATC — scenario-specific values (e.g. "email matches the signup payload") belong in the test file, not in the ATC. | HIGH | | A-A4 | Error ATCs assert both the status code and the canonical error-contract shape (`errorCode`, `message`, `details[]` or equivalent). | HIGH | | A-A5 | `expect` comes from `@playwright/test`, not from a custom assertion library. | MEDIUM | ### 3.4 Status-code coverage per endpoint For every endpoint the component exposes, the test file (or sibling test files) must cover the realistic status codes. Missing coverage is not a CRITICAL per-file reject but gates the handoff — see §4. | Method | Minimum expected coverage | |--------|---------------------------| | POST | 201 success + 400 validation + 401 unauthenticated (if protected) + 409 conflict (if applicable) | | GET | 200 success + 404 not found + 401 unauthenticated (if protected) | | PUT / PATCH | 200/204 success + 400 validation + 404 not found + 401 unauthenticated | | DELETE | 204 success + 404 not found + 401 unauthenticated | > Status codes are the AUTH/protocol layer of risk-beyond-AC. They are necessary but **not sufficient** — §3.4.1 (data boundaries) and §3.4.2 (state/temporal) cover the rest. Full canon: `agentic-qa-core/references/test-design-doctrine.md`. ### 3.4.1 Input-domain coverage (EP + BVA) — gates handoff | ID | Check | Severity | |----|-------|----------| | A-B1 | Each request field with a **range / limit / length** has boundary ATCs: `min-1·min·min+1` and `max-1·max·max+1` (parameterized). EP-merge does NOT satisfy this. | HIGH | | A-B2 | Empty / null / missing-required-field cases produce the expected 400 (one ATC per distinct invalid partition, not one lump "invalid" test). | HIGH | | A-B3 | Type/format violations (string where number, malformed date, oversized payload, unicode/emoji) are covered where the field accepts free input. | MEDIUM | | A-B4 | If no field has a range/limit, this section is explicitly marked N/A in the plan — not silently skipped. | LOW | ### 3.4.2 State-transition & temporal coverage — gates handoff | ID | Check | Severity | |----|-------|----------| | A-S1 | Stateful entities (status/lifecycle) have an ATC per valid transition AND per invalid transition (trigger fired in a state that should reject it). | HIGH | | A-S2 | Idempotency / double-submit is exercised for mutating endpoints that must be safe to retry. | MEDIUM | | A-S3 | Concurrency (two writers on the same resource → expected 409 / last-write-wins) is covered where the AC or domain implies it. | MEDIUM | | A-S4 | Timeout / retry / partial-failure rollback is covered for flows with external dependencies. | MEDIUM | ### 3.5 Error-contract assertions | ID | Check | Severity | |----|-------|----------| | A-E1 | 400 tests assert a validation error shape (field-level error list if the API provides one). | HIGH | | A-E2 | 401 tests call `api.clearAuthToken()` (or equivalent) before the request. | HIGH | | A-E3 | 404 tests use an unambiguously non-existent identifier (a fresh UUID, not a magic "999"). | MEDIUM | | A-E4 | 409 tests create the prior-state resource via a separate ATC (not via a direct DB write that bypasses business rules). | MEDIUM | | A-E5 | 5xx responses, when expected, are treated as contract (stability tests) — otherwise the test should fail loudly, not swallow them. | HIGH | ### 3.6 Token / auth propagation | ID | Check | Severity | |----|-------|----------| | A-T1 | `ApiFixture` overrides `setAuthToken` and `clearAuthToken` and forwards BOTH to every child API component. A new component missing either wire is a reject: it keeps a stale token after the fixture cleared one. The request context is NOT forwarded — it arrives through the constructor (`new XApi(options)`), so there is no `setRequestContext` to check. | CRITICAL | | A-T2 | Tests requiring auth call an auth ATC in `beforeEach` — not inline in every test. | HIGH | | A-T3 | Credentials come from `@variables` (resolved from `.env`). No hardcoded emails/passwords. | CRITICAL | | A-T4 | Token rotation / refresh flows are exercised at least once for components that own them. | MEDIUM | ### 3.7 Request / response schema validation | ID | Check | Severity | |----|-------|----------| | A-R1 | Request payloads are typed — no `any`, no partial objects cast to the payload type. | HIGH | | A-R2 | Response body is asserted against the declared type structurally (not just "is truthy"). | HIGH | | A-R3 | Pagination shape is asserted where lists are returned (`items`, `total`, `page` or the project's canonical envelope). | MEDIUM | | A-R4 | Dates and UUIDs are validated with regex or Playwright matchers, not just `expect(x).toBeDefined()`. | LOW | | A-R5 | Allure request/response attachments are produced by `ApiBase` automatically — no manual `testInfo.attach()` calls littering the ATC. | LOW | ### 3.8 Helper vs ATC discipline (API-specific) | ID | Check | Severity | |----|-------|----------| | A-D1 | Read-only GET used as a precondition is a plain helper (no decorator or `@step`), not `@atc`. A GET that is the subject of the test is an `@atc`. | HIGH | | A-D2 | Data-seeding helpers that orchestrate multiple resources live under `tests/components/steps/`, not inside an API component. | HIGH | | A-D3 | `api.data.*` factories generate payloads. No hand-built payload literals in the test body. | MEDIUM | --- ## 4. Handoff checklist — "ready for CI" Before marking the ticket complete and opening the PR, **every** box below must be true. A single unchecked item means NEEDS REVISION. ### 4.1 Code state - [ ] Shared checklist (`automation-standards.md` §10) — zero CRITICAL, zero HIGH. - [ ] Applicable delta (§2 E2E or §3 API) — zero CRITICAL, zero HIGH. - [ ] `bun run test <path>` — green, zero retries used. - [ ] `bun run types:check` — no errors. - [ ] `bun run lint:check` — no errors. - [ ] Component registered in `ApiFixture` / `UiFixture` as appropriate (Steps classes are instantiated directly in tests — no fixture registration). - [ ] Every `@atc('X')` resolves to a real TMS ticket (spot-check 2 random IDs). ### 4.2 Test coverage - [ ] Happy path covered. - [ ] All realistic error cases from §3.4 covered (API) or all primary UI error states covered (E2E): empty state, loading state, error banner, disabled CTA. - [ ] Input-domain boundaries (§3.4.1, EP + BVA) covered or explicitly N/A. - [ ] State transitions + temporal/concurrency risks (§3.4.2) covered or explicitly N/A. - [ ] Coverage exceeds the AC floor: risk-beyond-AC cases present (not just "every AC has a green test"). - [ ] Auth-failure test present if the feature is behind auth. - [ ] At least one test tagged `@critical` or `@smoke` if the feature is in the release-blocking set. ### 4.3 Data and environment - [ ] No hardcoded credentials — `@variables` / `.env` only. - [ ] No hardcoded production-like IDs — Faker or `api.data.*`. - [ ] `.env.example` updated if new variables were introduced. - [ ] `bun run api:sync` run if the OpenAPI contract changed. ### 4.4 Traceability - [ ] Ticket-ID prefix present in both `describe` and every `test()`. - [ ] `@atc` decorators present on every state-changing ATC. - [ ] `bun run kata:manifest` — component and ATCs appear with expected IDs. - [ ] TMS test case is linked (either via `@atc` ID already in TMS, or a follow-up documented in the PR body). ### 4.5 Hygiene - [ ] No `test.only`, no `test.skip`, no leftover `console.log` debugging. - [ ] No commented-out code or placeholder TODOs without an owner. - [ ] Commit message follows semantic prefix (`test:`, `feat:`, `fix:`, `refactor:`) and has no AI attribution lines. ### 4.6 CI preflight (for the PR) - [ ] File paths align with the project layout (`tests/e2e/{module}/...` or `tests/integration/{module}/...`). - [ ] Tests do not depend on local-only services that CI lacks (or such dependencies are documented). - [ ] Project-level config (`playwright.config.ts` projects array) already covers the folder the new tests live in — otherwise the tests will not run in CI. When every box is checked, the ticket is handed over to CI via the standard PR flow. If CI fails, return to Phase 2 Code; do not patch the PR with new conventions mid-review. --- ## Appendix · Legacy code cross-reference For PR comments that reference the legacy boilerplate's flat check IDs (`.prompts/stage-5-automation/review/*`). The current refactor split the 29+ flat checks into a shared list (`automation-standards.md` §10) plus deltas (this file). Use this table to resolve historical references. | Legacy code | Scope | New location | |---|---|---| | K-01 | KATA — component extends `UiBase`/`ApiBase` | `automation-standards.md` §10 / Component review | | K-02 | KATA — no direct Playwright imports in components | `automation-standards.md` §10 / Component review | | K-03 | KATA — imports via aliases (`@api/`, `@schemas/`, `@utils/`) | `automation-standards.md` §10 / Component review | | K-04 | KATA — ATCs return tuples or meaningful values | `review-checklists.md` §3.2 (A-H) + §3.2.1 | | K-05 | KATA — `@atc('ID')` tags present on state-changing methods | `automation-standards.md` §10 / ATC review | | K-06 | KATA — Steps module for reusable chains, not ATC-to-ATC calls | `automation-standards.md` §10 / Test file review | | K-07 | KATA — fixture selection (`{ api }` / `{ ui }` / `{ test }`) | `review-checklists.md` §2.4 + §3.6 | | K-08 | KATA — `TestContext` usage (config, faker) | `automation-standards.md` §10 / Component review | | K-09 | KATA — no duplicated helpers between components | `automation-standards.md` §10 / Component review | | A-01 … A-08 | ATC rules (atomicity, max 2 positional params, fixed vs test-level assertions, Equivalence Partitioning) | `automation-standards.md` §10 / ATC review | | T-01 … T-05 | TypeScript rules (parameter count, inline locators, alias imports, interface placement, silent-fail utilities) | `test-automation/references/typescript-patterns.md` | **Collision note**: the current local codes `A-xx` in §3 (API deltas: A-O, A-H, A-A, A-E, A-T, A-R, A-D) share a prefix with the legacy `A-01..A-08` (ATC rules) but have a different scope. Always read the containing section heading — the legacy meaning is the ATC-rules set under *shared* review (`automation-standards.md` §10), not the API delta. -
test-data-management.md 16 KB
# Test Data Management Strategy and mechanics for supplying test data in a KATA project. Load when designing preconditions, writing `beforeAll` / `beforeEach` hooks, building `DataFactory` generators, adding fixture JSON, or deciding how a new test should obtain its data. Architectural rules (no shared state, credentials from `.env`, one generator per test) are stated once in `automation-standards.md` and are not repeated here. This reference covers strategy, placement, and mechanics. --- ## 1. Golden Rule Never hardcode test data. Every test obtains its data at runtime — by discovering, modifying, or generating it. Hardcoded IDs, emails, or timestamps break the first time the environment changes. **Exception** — login credentials for pre-existing users come from `.env` (`LOCAL_USER_EMAIL`, `STAGING_USER_EMAIL`, and matching passwords). Everything else is dynamic. --- ## 2. Strategy — Discover → Modify → Generate Every precondition data need resolves to one of three patterns. Classify during the planning phase, not during coding. | Priority | Pattern | What it does | When to pick it | |----------|---------|-------------|-----------------| | 1 | **Discover** | Query the system (API or DB) for an entity already in the required state | Entity already exists in the environment in the desired state. Default. | | 2 | **Modify** | Find existing data, then mutate it via API into the required state | Entity exists but not in the right state, API supports the mutation | | 3 | **Generate** | Create new data from scratch via API (or DB fallback) | No usable data exists, full CRUD is available, or the test mutates state | Discover is preferred — zero impact on the database, realistic data. Modify adds one mutation. Generate is the last resort because it pollutes the environment. ### Override — when Generate beats Discover Use Generate even if data could be discovered when the test **mutates** state (POST, PUT, DELETE). Sharing a discovered entity across mutating tests creates order dependencies. Pair with `beforeEach` for isolation. ### Feasibility check (during planning) Before accepting a TC as an automation candidate, answer in order: 1. Can the entity be **discovered** reliably? (query DB or GET endpoint) 2. If not, can existing data be **modified** to reach the required state? (PUT/PATCH endpoint available?) 3. If not, can data be **created from scratch** via POST? 4. If none of the above → flag as blocker. The test is not yet an automation candidate; escalate to backend/infra. This check belongs in the Implementation Plan (Data Discovery step), never during coding. ### Decision flowchart ``` Need data for entity X? ├─ Exists in required state? ──── YES → Discover ├─ Exists but wrong state? ──── YES → Modify ├─ Can be created via API? ──── YES → Generate └─ None of the above? ──── BLOCKER (raise) ``` --- ## 3. DataFactory — central generator Single static class at `tests/data/DataFactory.ts`. Propagated through `TestContext` so both API and UI components reach it as `this.data`. ### Access patterns ```typescript // From a Layer 3 component (extends TestContext via base) const user = this.data.createUser(); // From a test file via fixture const user = ui.data.createUser(); const order = api.data.createOrder({ quantity: 2 }); // Direct import when no context is available import { DataFactory } from '@DataFactory'; DataFactory.createUser(); ``` ### Seeded generators (baseline) | Method | Returns | Purpose | |--------|---------|---------| | `createUser(overrides?)` | `TestUser` | Full user payload (email, password, name, first/last) | | `createCredentials(overrides?)` | `TestCredentials` | Email + password only (for login) | | `createTestId(prefix?)` | `string` | Unique identifier for tagging/tracing | | `createProduct(overrides?)` | `TestProduct` | Product domain payload | | `createOrder(overrides?)` | `TestOrder` | Order domain payload | Types live in `tests/data/types.ts`: ```typescript interface TestUser { email: string; password: string; name: string; firstName?: string; lastName?: string; } interface TestCredentials { email: string; password: string; } interface TestProduct { name: string; sku?: string; price?: number; categoryId?: number; } interface TestOrder { referenceNumber: string; productId: number; quantity: number; totalAmount: number; createdAt: string; } ``` ### Extending DataFactory Add a method in `DataFactory.ts` and a matching interface in `types.ts`. Generator rules: - Every field uses `faker` or a deterministic random helper. No hardcoded values except domain enums. - Accept `overrides?: Partial<T>` as the last positional arg. Spread it after defaults so callers win. - Use prefixes that identify test-origin data: `test.`, `CONF-`, `ORD-`. Makes DB cleanup and log inspection trivial. ```typescript static createCategory(overrides?: Partial<TestCategory>): TestCategory { return { name: `Category ${faker.commerce.department()} ${faker.number.int({ min: 1, max: 999 })}`, parentId: faker.number.int({ min: 1, max: 1000 }), createdAt: faker.date.recent().toISOString(), productCount: faker.number.int({ min: 0, max: 500 }), ...overrides, }; } ``` ### Never import `faker` directly in tests or components Always go through DataFactory. Direct `faker` imports bypass type contracts and spread random-data logic across the codebase. --- ## 4. Usage patterns ### Full object ```typescript const user = this.data.createUser(); // → { email: 'test.john.x7k2m9@example.com', password: 'TestAb3kL9mN!', name: 'John Doe', ... } ``` ### With overrides ```typescript const admin = this.data.createUser({ email: 'admin@example.com', name: 'Admin User' }); ``` ### Credentials only ```typescript const creds = this.data.createCredentials(); await ui.login.loginWithValidCredentials(creds); ``` ### Traceable ID ```typescript const testId = this.data.createTestId('order'); // → 'order-1707312000000-x7k2m9' ``` --- ## 5. DataFactory in components and tests ### Inside an ATC ```typescript @atc('ORDER-API-001') async createOrderSuccessfully(overrides?: Partial<TestOrder>) { const order = this.data.createOrder(overrides); const [response, body, sent] = await this.apiPOST<OrderResponse, TestOrder>('/orders', order); expect(response.status()).toBe(201); return [response, body, sent]; } ``` ### Inside a UI ATC ```typescript @atc('REG-UI-001') async registerNewUser(overrides?: Partial<TestUser>): Promise<TestUser> { const user = this.data.createUser(overrides); await this.page.locator('[data-testid="email"]').fill(user.email); await this.page.locator('[data-testid="password"]').fill(user.password); await this.page.locator('[data-testid="name"]').fill(user.name); await this.page.locator('[data-testid="submit"]').click(); await expect(this.page).toHaveURL(/.*dashboard.*/); return user; } ``` ### Inside a test ```typescript test('TICKET-ID: should register user with custom email', async ({ ui }) => { const user = ui.data.createUser({ email: 'vip@example.com' }); await ui.registration.registerNewUser(user); }); ``` --- ## 6. Precondition placement — `beforeAll` vs `beforeEach` Where you put the setup determines test speed AND test isolation. Pick by asking "does the test mutate the data?" | Hook | Pick when | Why | |------|-----------|-----| | `beforeAll` | Data is **read-only** — tests observe but never mutate | Query runs once, not N times. Fast. | | `beforeAll` | Setup is **expensive and shared** (login, heavy seeding) | Avoid repeating costly calls | | `beforeEach` | Data is **mutated** by each test (POST, PUT, DELETE) | Each test needs a fresh isolated state | | `beforeEach` | Setup is **cheap and must stay isolated** (page navigation) | Ensures independence even if one test fails | Rule of thumb: test **reads** → `beforeAll`. Test **writes** → `beforeEach`. ### Passing data from setup to tests Declare variables at the `describe` scope. `beforeAll` populates them without assertions; each test validates its own precondition with `test.skip()`. ```typescript test.describe('Orders: page states', () => { let completedOrder: OrderCandidate | null; let pendingOrder: OrderCandidate | null; test.beforeAll(async ({ api }) => { // DISCOVER — no assertions here completedOrder = await api.orders.findOrderWithState('completed'); pendingOrder = await api.orders.findOrderWithState('pending'); }); test('TICKET-ID: should show details for completed order', async ({ ui }) => { if (!completedOrder) return test.skip(true, 'No completed order available'); await ui.orders.selectOrder({ orderId: completedOrder.id }); }); test('TICKET-ID: should show pending state', async ({ ui }) => { if (!pendingOrder) return test.skip(true, 'No pending order available'); await ui.orders.selectOrder({ orderId: pendingOrder.id }); }); }); ``` --- ## 7. Never `expect` inside `beforeAll` If a `beforeAll` assertion fails, ALL tests in the describe block fail — including unrelated ones. That hides what really broke. ```typescript // WRONG — one missing precondition kills every test in the block test.beforeAll(async ({ api }) => { pendingOrder = await api.orders.findByState('pending'); expect(pendingOrder, 'No pending order').toBeDefined(); // blocks unrelated tests }); // WRONG — cryptic null access in the test test('should display pending', async ({ ui }) => { await ui.orders.selectOrder(pendingOrder.id); // TypeError: Cannot read 'id' of null }); // RIGHT — beforeAll discovers only, each test skips with a message test.beforeAll(async ({ api }) => { pendingOrder = await api.orders.findByState('pending'); }); test('TICKET-ID: should display pending', async ({ ui }) => { if (!pendingOrder) return test.skip(true, 'No pending order available'); await ui.orders.selectOrder(pendingOrder.id); }); ``` Why: - `beforeAll` is shared setup — it must not contain assertions that block unrelated tests. - Each test validates its own precondition with `test.skip(true, reason)`. - The report distinguishes "skipped — missing data" from "failed — actual bug". --- ## 8. Cleanup — `afterAll` and `afterEach` If setup **modifies** or **generates** data, restore or delete it so the environment does not accumulate residue. | Hook | When to use | |------|-------------| | `afterEach` | Each test mutated data independently — restore after each | | `afterAll` | One shared mutation in `beforeAll` — restore once at end | ```typescript test.describe('TICKET-ID: Validate order status actions', () => { let originalStatus: string; test.beforeAll(async ({ api }) => { const [, order] = await api.orders.getStatus(orderId); originalStatus = order.status; await api.orders.resetToProcessing(orderId); }); test.afterAll(async ({ api }) => { await api.orders.setStatus(orderId, originalStatus); }); }); ``` | Pattern used in setup | Cleanup required | |-----------------------|------------------| | Discover | None — data was only read | | Modify | Restore original state in `afterAll` / `afterEach` | | Generate | Delete created entity in `afterEach` (or accept the leak if the env is disposable) | --- ## 9. Placement summary | Data strategy | Hook | Variable scope | Cleanup | |---------------|------|----------------|---------| | Discover | `beforeAll` (query once) | describe scope | None | | Modify | `beforeAll` or `beforeEach` | describe scope | `afterAll` / `afterEach` to restore | | Generate | `beforeEach` (isolated per test) | describe scope or inline | `afterEach` to delete (if env needs it) | --- ## 10. Static fixture files For reference data that does not change — roles, permission matrices, mock response bodies, configuration trees — use `tests/data/fixtures/` JSON (or CSV if tabular). | Fixtures (`tests/data/fixtures/`) | DataFactory | |-----------------------------------|-------------| | Fixed roles / permissions | Test users | | Reference catalogs | Transactional data | | API mock responses | Request payloads | | Configuration trees | Domain objects with business logic | ### JSON example ```json // tests/data/fixtures/roles.json { "admin": { "name": "Administrator", "permissions": ["read", "write", "delete", "admin"] }, "catalog_manager": { "name": "Catalog Manager", "permissions": ["read", "write", "publish"] }, "viewer": { "name": "Viewer", "permissions": ["read"] } } ``` ### Usage ```typescript import roles from '@data/fixtures/roles.json'; test('TICKET-ID: admin can delete', async ({ api }) => { const user = api.data.createUser(); await api.users.assignRole(user.id, roles.admin); }); ``` ### Parameterised tests from JSON ```typescript import usersData from '@data/fixtures/users.json'; for (const user of usersData) { test(`TICKET-ID: should signup ${user.type} user`, async ({ ui }) => { await ui.signup.signupWithValidCredentials({ email: user.email, password: user.password }); }); } ``` Commit `tests/data/fixtures/` and `tests/data/uploads/`. Gitignore `tests/data/downloads/` and `test-results/`. --- ## 11. Isolation & parallelism The shipped config runs a single worker (`playwright.config.ts: workers: 1`), but uniqueness still matters — for reruns against the same environment and for the future parallelism upgrade. DataFactory guarantees it by combining a prefix, an epoch-ms timestamp, and a 6-char random suffix: ``` Email: test.john.x7k2m9@example.com TestId: order-1707312000000-x7k2m9 ``` That is enough to avoid collisions across reruns AND across workers once parallelism is enabled. Do not design tests that rely on sequential data (e.g., "the first order in the DB"); always tag with a unique marker first. For stabilization rules (no hardcoded waits, no retries, specific-condition waiting), see `automation-standards.md` §7. --- ## 12. Credentials & sensitive data Credentials for pre-existing users come from environment variables. Never hardcode. Never check into git. ```typescript // config/variables.ts export const config = { testUser: { email: process.env.LOCAL_USER_EMAIL!, password: process.env.LOCAL_USER_PASSWORD!, }, }; // usage const { email, password } = api.config.testUser; await api.auth.loginSuccessfully({ email, password }); ``` `.env` keys by environment: ``` LOCAL_USER_EMAIL=test@example.com LOCAL_USER_PASSWORD=SecurePassword123! STAGING_USER_EMAIL=staging@example.com STAGING_USER_PASSWORD=StagingPassword123! ``` Decorators (`@atc`, `@step`) automatically mask parameters named `password`, `token`, `secret` in trace output. Keep the names canonical so masking works. --- ## 13. DO / DON'T ### DO - Use `this.data.createX()` inside components, `ui.data.createX()` / `api.data.createX()` in tests. - Classify data as Discover / Modify / Generate during planning. - Declare describe-scope variables and let `beforeAll` populate them. - Guard every test that depends on discovered data with `test.skip()`. - Give generated entities identifiable prefixes (`test.`, `CONF-`, `ORD-`). - Clean up modified data in `afterAll`; delete generated data in `afterEach` when the env matters. ### DON'T - Hardcode emails, IDs, reference numbers, or dates. - Put `expect` inside `beforeAll`. - Share generated data between tests. - Import `faker` directly — go through DataFactory. - Use production data. - Create generators without matching TypeScript interfaces. - Mutate data discovered by another test without restoring it. --- ## 14. Quick reference ```typescript // From components this.data.createUser(); this.data.createCredentials(); this.data.createTestId('prefix'); this.data.createProduct(); this.data.createOrder(); // From tests ui.data.createUser(); api.data.createOrder({ productId: 42, quantity: 2 }); // Direct import { DataFactory } from '@DataFactory'; DataFactory.createUser(); // With overrides this.data.createOrder({ totalAmount: 500 }); ``` File shapes: ``` tests/data/DataFactory.ts # generators tests/data/types.ts # TestUser, TestOrder, TestCategory, ... tests/data/fixtures/*.json # reference data (committed) tests/data/uploads/* # upload inputs (committed) tests/data/downloads/* # download destination (gitignored) ``` -
typescript-patterns.md 15.4 KB
# TypeScript Patterns for KATA Coding conventions that apply to every Layer 2, 3, and 3.5 file. Load when writing or reviewing component code. For architectural rules (layers, ATC identity, fixture selection), see `kata-architecture.md`. --- ## 1. Parameter Pattern (Max 2 Positional) If a function has 3+ parameters, use an object parameter. No exceptions. ```typescript // WRONG function interceptResponse( page: Page, urlPattern: string, action: () => Promise<void>, timeout?: number, attachToAllure?: boolean, ) { ... } // RIGHT interface InterceptArgs { urlPattern: string | RegExp; action: () => Promise<void>; timeout?: number; attachToAllure?: boolean; } function interceptResponse(args: InterceptArgs) { ... } ``` Benefits: self-documenting call sites, IDE autocomplete shows parameter names, order-independent, easy to add optional params without breaking changes. This rule applies to ATCs as well. An ATC that took `(email, password, rememberMe, redirectUrl)` must take `(args: LoginArgs)` instead. --- ## 2. DRY by Layer — Where Does Code Live? The physical location of helper code matters. Moving a helper to the wrong place creates coupling and fragile tests. | Location | When to put code here | Example | |----------|-----------------------|---------| | `tests/utils/` | Agnostic utility (works for API AND UI, no Playwright deps) | `allure.ts` (Allure attachment helpers), string formatters | | `UiBase` | Requires `PageContext` (Playwright `page`) | `interceptResponse()`, `waitForApiResponse()` | | `ApiBase` | Requires `APIRequestContext` | `apiGET()`, `apiPOST()` | | `TestContext` | Shared across API + UI, no external Playwright deps | `faker`, `config` accessors, environment selection | | Layer 3 component | Domain-specific logic for one resource/page | `generateOrderPayload()` inside `OrdersApi` | Architectural principle: > Anything that needs `PageContext` goes in UiBase. > Anything that needs `APIRequestContext` goes in ApiBase. > Only truly agnostic utilities go in `tests/utils/`. If you are tempted to put an API helper in `tests/utils/`, stop — it means the utility still depends on `APIRequestContext`, which means it belongs in `ApiBase`. --- ## 3. Shared Locator Pattern (UI) If a locator is used in 2+ ATCs of the same component, extract it to a class property. If used once, keep it inline. ```typescript class CheckoutPage extends UiBase { // Arrow function for dynamic locators private readonly productRow = (name: string) => this.page.locator(`[data-product="${name}"]`); // Static element used in multiple ATCs private readonly submitButton = () => this.page.locator('button[type="submit"]'); @atc('TICKET-ID') async addProductSuccessfully(product: string) { await this.productRow(product).click(); await this.submitButton().click(); } @atc('TICKET-ID') async removeProductSuccessfully(product: string) { await this.productRow(product).locator('[data-action="remove"]').click(); } } ``` When to extract: - Used in 2+ ATCs of the same component. - Complex selector with fallbacks. - Dynamic selector that takes parameters. When to keep inline: - Used only once. - Simple, obvious selector like `button[type="submit"]` used once. When to move to UiBase: - Used across **multiple components** (rare) — e.g., a global nav element. Never extract to a separate `locators/*.ts` file. That is an anti-pattern in KATA. --- ## 4. Type Definitions Define interfaces at the top of the file, after imports, before the class. ```typescript import type { Page } from '@playwright/test'; // ============================================ // Types // ============================================ export interface InterceptedData<TRequest = unknown, TResponse = unknown> { url: string; method: string; status: number; requestBody: TRequest | null; responseBody: TResponse | null; } export interface InterceptResponseArgs { urlPattern: string | RegExp; action: () => Promise<void>; timeout?: number; } // ============================================ // Implementation // ============================================ export class UiBase { ... } ``` Every payload and response type used by a component should be defined here (or imported from `@schemas/{domain}.types` if generated from OpenAPI). Avoid inline object types in method signatures — always name the interface. --- ## 5. Generic Type Parameters Use descriptive generic names and default to `unknown`. ```typescript // GOOD — clear intent with defaults async interceptResponse<TRequest = unknown, TResponse = unknown>( args: InterceptResponseArgs, ): Promise<InterceptedData<TRequest, TResponse>> { ... } // Usage with types const { responseBody } = await this.interceptResponse<LoginPayload, TokenResponse>({ urlPattern: /\/auth\/login/, action: async () => await this.submitButton().click(), }); // Usage without types (falls back to unknown — safe) const { responseBody } = await this.interceptResponse({ urlPattern: /\/api\/data/, action: async () => await this.loadButton().click(), }); ``` Never use single-letter generics (`T`, `U`) in public APIs — they give no hint about intent. Internal private methods may use `T` if obvious. --- ## 6. Private vs Public Methods Private helpers live only inside the class; public methods form the external API. ```typescript class UiBase extends TestContext { // PUBLIC — part of the class API, documented async interceptResponse<T>(args: InterceptResponseArgs): Promise<T> { const response = await this.waitForMatchingResponse(args.urlPattern); return this.parseResponseBody(response); } // PRIVATE — internal helper private matchesPattern(url: string, pattern: string | RegExp): boolean { if (pattern instanceof RegExp) return pattern.test(url); const regexPattern = pattern.replace(/\*\*/g, '.*').replace(/\*/g, '[^/]*'); return new RegExp(regexPattern).test(url); } private async parseResponseBody<T>(response: Response): Promise<T | null> { try { return (await response.json()) as T; } catch { return null; } } } ``` Do not mark a method `public` if it is a helper inside an ATC. Keep private methods private — ATCs stay the public API surface. --- ## 7. Error Handling Fail fast in the methods a test can reach with a descriptive error. Silent fail in utility helpers that legitimately may not find data. ```typescript // REACHABLE BY A TEST (an ATC) — fail fast @atc('TICKET-ID') async createOrderSuccessfully(payload: OrderPayload): Promise<[APIResponse, OrderResponse, OrderPayload]> { const [response, body, sent] = await this.apiPOST<OrderResponse, OrderPayload>('/orders', payload); expect(response.status()).toBe(201); return [response, body, sent]; } // The HTTP helpers themselves are `protected` (see kata-architecture.md §4): // a test must go through an ATC, never past it to raw HTTP. `ApiBase` throws // from its `get request()` getter when no context is available — there is no // `setRequestContext()` to call, the context is constructor-injected. // UTILITY — silent fail, return null/undefined private async parseResponseBody<T>(response: Response): Promise<T | null> { try { return (await response.json()) as T; } catch { return null; } // response may not be JSON — that is not an error } ``` Rule: | Method type | On unexpected state | Rationale | |-------------|---------------------|-----------| | Public ATC or helper | Throw with descriptive message | Test should fail loudly at the call site | | Private utility (parser, matcher) | Return `null` / `undefined` | Caller decides what to do with missing data | Never swallow errors inside ATCs without re-throwing — the test must fail visibly. --- ## 8. Import Organization Group imports in order: type imports, external value imports, internal value imports. One blank line between groups. ```typescript // 1. Type imports (use 'import type') import type { Environment } from '@variables'; import type { Page, Request, Response } from '@playwright/test'; // 2. Value imports from external packages import { expect } from '@playwright/test'; // 3. Value imports from internal modules import { TestContext } from '@TestContext'; import { buildUrl, config } from '@variables'; import { attachRequestResponseToAllure } from '@utils/allure'; ``` Always prefer `import type` for types that are only referenced in type positions — it erases cleanly at build time. ### Import aliases are mandatory No relative imports. Configure once in `tsconfig.json`: ```json { "compilerOptions": { "paths": { "@/*": ["./*"], "@ui/*": ["./tests/components/ui/*"], "@api/*": ["./tests/components/api/*"], "@steps/*": ["./tests/components/steps/*"], "@utils/*": ["./tests/utils/*"], "@data/*": ["./tests/data/*"], "@variables": ["./config/variables.ts"], "@TestContext": ["./tests/components/TestContext.ts"], "@UiFixture": ["./tests/components/UiFixture.ts"], "@ApiFixture": ["./tests/components/ApiFixture.ts"], "@TestFixture": ["./tests/components/TestFixture.ts"], "@DataFactory": ["./tests/data/DataFactory.ts"], "@openapi": ["./api/openapi-types.ts"], "@schemas/*": ["./api/schemas/*"], "@schemas": ["./api/schemas/index.ts"] } } } ``` ```typescript // RIGHT import { config } from '@variables'; import { UsersApi } from '@api/UsersApi'; // WRONG import { config } from '../../../config/variables'; ``` Lint rejects relative imports: `KATA_IMPORT_ALIASES` in `eslint.config.base.js`, a core `no-restricted-imports` block scoped to `tests/**/*.ts` + `playwright.config.ts`. There is no `eslint-plugin-import` in this repo. Dynamic `await import('./x')` is outside the rule. Fix them — do not disable the rule. --- ## 9. Test Design Conventions (code-level) These conventions apply to every test file. They are the code-level expression of the TC-identity rule (documented in `kata-architecture.md`). ### Helpers live at the top of the component ``` OrdersApi.ts: // --- Helpers (no @atc) --- getOrders(filters) → GET /orders getOrderById(id) → GET /orders/{id} getTotals(filters) → GET /orders/totals // --- ATCs (@atc) --- @atc('TICKET-ID') createOrderSuccessfully(orderData) → POST /orders + GET verification ``` Helpers can be called inside ATCs as verification steps, inside test files for preconditions, or inside Steps for setup chains. ### Test file naming Pattern: `{verb}{Feature}.test.ts` in camelCase with a verb describing the user action. - Good: `applyDiscount.test.ts`, `createOrder.test.ts`, `refreshCatalog.test.ts`, `authenticateUser.test.ts`. - Bad: `discount.test.ts` (no verb), `orderFlow.test.ts` (too broad), `check_discount.test.ts` (verb is a test verb, not a user action). ### Test block naming ```typescript test.describe('TICKET-ID: Validate discount codes', () => { test('TICKET-ID: should apply percentage discount when code is valid', ...); test('TICKET-ID: should apply fixed-amount discount when code is valid', ...); test('TICKET-ID: should reject discount when code has expired', ...); }); ``` - `describe` may include the ticket ID when the file is tied to a single ticket. - Every `test()` includes the ticket ID as a prefix followed by `should {behavior} when {condition}`. - One ticket per `describe`; never split a ticket across files. Multiple tickets can coexist in one file when they test different aspects of the same feature. ### Tests validate FLOWS, not properties One test validates a complete flow with multiple assertions. Do NOT split six tests that each check one field of the same response. ```typescript // WRONG — splits a single flow into six tests test('should return orders', async ({ api }) => { ... }); test('should have referenceNumber', async ({ api }) => { ... }); test('should have totalAmount', async ({ api }) => { ... }); // RIGHT — one test, complete contract test('TICKET-ID: should create order with correct totals when discount applied', async ({ api }) => { const [, order] = await api.orders.createOrderSuccessfully(orderData); const totals = await api.orders.getTotals({ orderId: order.id }); expect(order.id).toBeDefined(); expect(order.discountApplied).toBe(true); expect(totals.finalAmount).toBe(totals.baseAmount - totals.discountAmount); }); ``` Separate tests only when the scenario is fundamentally different: | Separate test? | Reason | |----------------|--------| | Yes | Different flow (positive vs negative) | | Yes | Different precondition that changes the outcome | | Yes | Different user role or permissions | | No | Same flow, different field assertions | | No | Same response, different property checks | ### Test independence - No shared state between tests. - Each test generates its own data via `TestContext` helpers or faker (e.g., `api.generateUserData()`). - Do not rely on test ordering. - Clean up in `afterEach` only when the test created state that would persist and affect others. ### Assertion layers ``` Test flow ├── ATC 1: createOrderSuccessfully() │ └── [ATC assertions: status 201, order persisted] ├── ATC 2: applyDiscountSuccessfully() │ └── [ATC assertions: discount applied, total recalculated] └── Test-level assertions: └── [Final state: final total = base - discount, tax applied correctly] ``` ATC assertions validate that each individual action worked. Test-level assertions validate the outcome of combining multiple actions. ### Preconditions strategy Each test sets up its own data via API (or DB) before executing the scenario: 1. Preconditions — prepare test data to create the scenario. 2. Actions — perform the test steps (API calls or UI interactions). 3. Assertions — validate expected behavior with the given data. Rules: each test creates its own scenario independently; tests must not depend on or interfere with others; preconditions come from API endpoints or the DB connection; shared environments are used collaboratively but data inside each test is managed per-test. ### Integration vs E2E test design | Aspect | Integration | E2E | |--------|-------------|-----| | Scope | API endpoint chain (2-3 endpoints) | Full user journey (UI + API) | | Speed | Fast (no browser) | Slower (browser required) | | Fixture | `{ api }` | `{ ui }` / `{ test }` | | Preconditions | API calls | API calls (setup) + UI (action) | | Value | Validates business logic correctness | Validates user experience | Both follow the same principles: complete flows, ATCs for actions and helpers for reads, multiple assertions per test, separate tests only for fundamentally different scenarios. --- ## 10. Quick Reference | Pattern | Rule | Example | |---------|------|---------| | Parameters | Max 2 positional, else use object | `fn(args: Args)` not `fn(a, b, c, d)` | | Utilities | Only agnostic go to `utils/` | `allure.ts` yes, `interception` no | | Locators | Extract if used 2+ times in same component | `private readonly btn = () => ...` | | Types | Define at top, after imports, named interfaces | `interface X { ... }` | | Generics | Descriptive names, default to `unknown` | `<TRequest = unknown>` | | Private | Internal helpers only | `private matchesPattern()` | | Errors | Public: fail fast; private utility: silent fail | `throw Error` vs `return null` | | Imports | Order: type, external, internal; aliases only | Grouped with blank lines | | Test files | `{verb}{Feature}.test.ts`, ticket ID in `test()` | `applyDiscount.test.ts` | | Test independence | No shared state, each test generates data | `api.generateUserData()` |
-
-
SKILL.md 48.4 KB
--- name: test-automation description: "Plan, write, and review automated tests following KATA (Komponent Action Test Architecture) on Playwright + TypeScript, or explain existing automated tests in a sealed read-only mode. Use when writing E2E or API/integration tests, creating Page or Api components, designing ATCs, parameterizing test data, registering fixtures, reviewing test code for KATA compliance, or requesting break-down-tests / a plain-English test breakdown. The explain mode reads source and reports assertions without entering Plan-Code-Review or editing tests. Do NOT use for running suites (regression-testing), documenting TCs in Jira/Xray (test-documentation), onboarding a repo (project-discovery), or orchestrating sprint-wide testing (sprint-testing)." license: MIT compatibility: [claude-code, copilot, cursor, codex, opencode] complementary_categories: [testing-e2e, testing-api, testing-component, automation-cli, accessibility] --- ## Forbidden invocations **NEVER invoke `/sdd-*` skills from this workflow.** SDD is an optional user-installed ceremony; this skill ships self-contained and does not chain SDD under any condition. If you need to refactor KATA, fixtures, cli/, scripts/, or api/schemas/ pipeline, exit this skill first and invoke `/framework-development` — which itself runs Plan → Code → Verify → Archive natively (no SDD required). This boundary is mechanical, not advisory: `scripts/lint-skills.ts` rejects any `/sdd-` mention outside this section. See: `.agents/skills/agentic-qa-core/references/skill-composition-strategy.md` §4 (governs users who manually install SDD). # Test Automation — Plan, Code, Review Produce KATA-compliant automated tests for an existing Playwright + TypeScript project. Three phases, always in this order: **Plan → Code → Review**. Never jump straight to code. KATA (Komponent Action Test Architecture) rewires the usual Page Object pattern. If you write tests the "standard" way, most will be rejected at review. Load the relevant reference before writing code in that area. --- ## Dependencies Requires `agentic-qa-core`. Loads on demand: - `agentic-qa-core/references/test-design-doctrine.md` — **MANDATORY before planning which ATCs to write from acceptance criteria.** Governs the 1:N ATC derivation, the formal-technique triggers (incl. BVA, which KATA's EP-merge rule does NOT replace), and the floor-not-ceiling coverage model. - `agentic-qa-core/references/briefing-template.md`, `agentic-qa-core/references/dispatch-patterns.md`, `agentic-qa-core/references/orchestration-doctrine.md`, `agentic-qa-core/references/session-management.md`, `agentic-qa-core/references/preflight-gate.md`, `agentic-qa-core/references/adr-doctrine.md` — cited inline by the sections that use them. ## Compact Rules **Test-design doctrine (binding — full canon: `agentic-qa-core/references/test-design-doctrine.md`):** - "All ACs covered" is the FLOOR, not the success bar. The ATC set must also cover risk-beyond-AC: invalid/boundary inputs, auth/error paths, state transitions, and anomalies the AC is silent on. - 1:N is the default: one AC maps to multiple ATCs. EP-merge collapses same-behavior inputs INSIDE one partition into a parameterized ATC — it must NEVER collapse across distinct partitions, boundaries, or states. BVA cases are required wherever a range/limit/length/date-window exists (EP alone misses off-by-one). - Apply techniques by trigger: EP always; BVA on ranges/limits; State-Transition for stateful flows; Decision Table when 2+ conditions interact; Pairwise when 3+ combinable factors (log the reduction). - Parametrize for artifact economy: same-behavior data variants → ONE parameterized `@atc` (fixture / data-factory rows iterated by the test) per partition, NOT N ATCs; split only when action / outcome / state differs. (Canon: doctrine §"Part 2.5".) - An AC is the business assertion; an ATC is its concrete exploration (Precondition + Action + Assertions). Run the Test-Design Checklist before finalizing the plan. **Test-automation operational rules:** - Plan → Code → Review, always in order. Only automate `Candidate` verdicts from `/test-documentation`. - Fixture selection: API-only → `{ api }` (no browser); UI-only → `{ ui }`; hybrid → `{ test }`. - ATC = atomic mini-flow; NEVER calls another ATC. Reusable chains → a Steps module. - Max 2 positional params (3+ → object param). Locators inline (extract only at 2+ uses). Imports via aliases (`@api/`, `@schemas/`, `@utils/`) — no relative imports. - Public methods fail fast; utilities silent-fail (return null). Validate against `kata-manifest.json` before adding components/ATCs (anti-duplication gate). **Read full SKILL.md when**: writing KATA component code, choosing fixtures for a hybrid flow, or applying the Phase 3 review checklist. --- ## Mode routing Resolve mode before any readiness preflight or session workflow. - `explain`: selected by the legacy `break-down-tests` alias or an explicit request to explain existing automated tests. Forward `$ARGUMENTS` unchanged, load only `references/explain-tests.md`, produce its read-only report, then stop. Do not create session state, run Plan -> Code -> Review, edit tests, regenerate `kata-manifest.json`, or call Jira/TMS. - `automate` (default): all normal KATA planning, coding, and review triggers. Continue with the workflow below. If the invocation could mean either explanation or implementation, ask which outcome is wanted. Never infer implementation from a read-only explanation request. --- ## Subagent Dispatch Strategy > **Orchestration & Session contracts**: this skill follows `agentic-qa-core/references/orchestration-doctrine.md` (mandatory subagent dispatch — main thread is command center) AND `agentic-qa-core/references/session-management.md` (Phase 0 resume check, plan-first persistence at `.session/<skill-slug>/<scope>/`, archive on completion). Phase 0 (resume check) and Phase 1 (plan write) are NOT optional. The orchestrator also applies the per-stage **Definition-of-Done gates** in `agentic-qa-core/references/stage-gates.md`: verify a stage's DoD (planning stages include the Test-Design Checklist) BEFORE recording its progress checkpoint and advancing. This skill is **per-scope**: `<scope>` = `<JIRA-KEY>` (ticket-driven / regression-driven) or `<module-slug>` (module-driven). Session state lives at `.session/test-automation/<scope>/{plan.md, progress.md}` per `agentic-qa-core/references/session-management.md` §3 + §9. The session `plan.md` is a thin INDEX that cites the canonical domain artifacts (`spec.md`, `automation-plan.md`, `atc/*.md`) under the Epic's `test-specs/` tree (`.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/`) — domain content stays in the existing PBI tree, not duplicated. This skill is compliant with the doctrine in `AGENTS.md` §"Orchestration Mode (Subagent Strategy)" and the session contract in `.agents/skills/agentic-qa-core/references/session-management.md`. Every dispatch follows the 7-component briefing format defined in `.agents/skills/agentic-qa-core/references/briefing-template.md`, and the pattern selected per phase matches the decision guide in `.agents/skills/agentic-qa-core/references/dispatch-patterns.md`. The Plan, Code, and Review phases each carry distinct context-isolation needs — Plan keeps KATA architectural reads out of the orchestrator, Code isolates multi-file edits, Review fans out three independent verifiers in parallel. | Stage | Pattern | Subagent role | |------------------------------------------------|----------------------|--------------------------------------------------------------------------------------------------------------------------------| | Plan (`spec.md` + `automation-plan.md`) | Single | one Plan subagent returns the two artifacts; protects orchestrator from KATA architectural reads | | Code (writing E2E or API tests) | Sequential | one Code subagent per scope (module = 1 subagent per TC; ticket = 1 subagent total); edits-many-files inside isolates context | | Review — `bun run test` | Parallel (sub-stage) | one Verifier subagent runs the test suite | | Review — `bun run types:check` | Parallel (sub-stage) | one Verifier subagent runs typecheck | | Review — `bun run lint:check` | Parallel (sub-stage) | one Verifier subagent runs lint | | Review aggregation + merge/reject decision | Single | inline — orchestrator reads the 3 Verifier reports and decides | - **Code phase scope rule**: each Code subagent edits multiple files in isolation, returns a list of changed files + a one-line summary per file. The orchestrator never reads the diffs — only the summary. If the user wants to see actual diffs, the orchestrator runs `git diff` inline after the subagent returns. - **On any Verifier failure**: STOP, return the failing report verbatim to the user, do NOT auto-fix the test code, do NOT re-dispatch the Code phase without user approval. See `.agents/skills/agentic-qa-core/references/orchestration-doctrine.md`. - **MANDATORY context doc for Plan + Code briefings**: include `kata-manifest.json` (root) in the "Context docs" component (item 2 of the 7-component briefing). Without it the subagent will scan `tests/components/**` directly, burn tokens, and risk proposing duplicates. See Critical Rule #12 in `AGENTS.md`. --- ## Inputs — read these first, in this order Canonical reading order for any AI starting cold on a test-automation workflow. Read in order; stop earlier when later inputs add no signal for the scope at hand. 1. `kata-manifest.json` (root) — authoritative registry of every Component (`api[]`, `ui[]`) and every `@atc('TICKET-ID')` ID. Anti-duplication gate per Critical Rule #12 in `AGENTS.md`. MUST load before proposing any new `Page`, `Api`, `Steps` module, or `@atc` ID. 2. `.agents/skills/test-automation/references/kata-architecture.md` + `.agents/skills/test-automation/references/typescript-patterns.md` — full doctrine for KATA layers (TestContext / Base / Domain / Fixture), ATC identity, fixture selection, import-alias rules, params contracts. 3. `tests/components/` — existing Api / Page / Steps shape on disk. Establishes naming, helper-vs-ATC split, fixture registration patterns to follow. 4. The Story's `implementation-plan.md` (dev plan) + the ATP under `.context/PBI/epics/EPIC-<KEY>-<slug>/stories/STORY-<KEY>-<slug>/` — Jira-synced, READ-ONLY caches. Jira is source of truth; NEVER hand-write these. Materialize via `bun run jira:sync-issues get <STORY-KEY> --include-comments`, then **read the ENTIRE synced Story folder** — every per-field `.md` (`story.md`, `acceptance-criteria.md`, scope, business rules, etc.) **plus `comments.md`** — not just one field. Omitting ACs, scope, business rules, or comment context produces incomplete ATCs. The **ATP read is modality-aware** (resolve via `.agents/project.yaml` `testing.tms_cli`, same gate as `/test-documentation` §Phase 0): - **Modality jira-native**: ATP = Story field `{{jira.acceptance_test_plan}}` → synced `acceptance-test-plan.md` in the Story folder (from the same `jira:sync-issues get <STORY-KEY> --include-comments`). - **Modality jira-xray**: ATP = Test Plan issue `description` → `bun run jira:sync-issues get <ATP_KEY>` → `test-plans/ATP-<KEY>-<slug>.md`; per-TC run results come from `[TMS_TOOL]` (xray-cli), not the sync. Filename note: the acronym prefix comes from a conforming ladder title; a Plan or Execution whose title does not follow the grammar keeps the legacy `TESTPLAN-` / `TESTEXEC-` / `RETESTEXEC-` prefix. The dev `implementation-plan.md` carries the implementation approach + (when produced by `/test-documentation`) the per-TC candidate verdict and component mapping. Cite from the session `plan.md` rather than duplicating. NOTE: this is the Story-folder *dev* `implementation-plan.md` — do NOT confuse it with the hand-authored *automation* plan (`test-specs/<scope>/automation-plan.md`) you write in Phase 1. 5. The Story's AC (acceptance criteria) — source of truth for scenarios that become ATCs. Read from the same synced `.md` files (`acceptance-criteria.md` / `story.md`) produced by `bun run jira:sync-issues get <STORY-KEY> --include-comments`. NEVER use `[ISSUE_TRACKER_TOOL]` `view` for these custom fields — `view` returns `null` for `customfield_*`. If a field is absent from the instance, the sync emits a pointer stub and the content lives in comments/description per `.agents/jira-required.yaml` `fallback:`. Resolve the issue key from the scope picker. **TC note**: a TC body = the `Test` issue `description` (synced both modalities via `bun run jira:sync-issues get <TEST-KEY>`); the Xray Gherkin / Test-Steps plugin field is NOT synced — it mirrors the description, so read the synced TC `.md` for Gherkin/steps. 6. `api/schemas/` — OpenAPI-derived TypeScript types. Refresh via `bun run api:sync` if stale. Required for any Api component touching a new endpoint. 7. `.env` — credentials (`LOCAL_USER_EMAIL`, `STAGING_USER_PASSWORD`, etc.) read via `config.testUser` from `@variables`. Never hardcode; never guess. 8. `agentic-qa-core/references/artifact-lifecycle.md` — the TC status ladder this skill owns (`candidate` → `in_automation` → `pull_request` → `automated`), the unmapped-status fallback (§4), and the light stage verifier that closes Review (§5). Read BEFORE firing any transition. --- ## Readiness Preflight Gate (MANDATORY — runs before Phase 0) > Full doctrine: `agentic-qa-core/references/preflight-gate.md`. Runs FIRST, before the resume check and scope picker. Two laws: (1) **args-as-answers** — the scope, ticket key, and "API test" vs "E2E test" are provided args; ask only the gaps. (2) **probe, don't assume**. Surface gaps + REDs as ONE `AskUserQuestion` checklist; self-fix with approval + explanation; STOP on any blocking RED. Note: this is distinct from the **anti-duplication** "Pre-flight checklist" inside Phase 1 (which cross-checks `kata-manifest.json` for reuse) — this gate is about tools + env being ready to write and run code. **Generic baseline** (env resolution, test-user creds, secret/restart handling, the two laws, output contract) is inherited from the reference §3.1 — not repeated here. Below is only this skill's **specific capability delta**. | Capability | Need | Why here | |---|---|---| | Framework adapted (artifacts present) | REQUIRED | Cannot write project ATCs against the generic `Example*` scaffolds the boilerplate ships. Probe the reference §4 ADAPTED signals; still generic → STOP and tell the user to run `/project-discovery` → `/adapt-framework` themselves. The gate NEVER auto-runs them. | | Dev toolchain | REQUIRED | The Review gate runs `bun run test` / `bun run types:check` / `bun run lint:check`. Resolve them at t=0, not at Phase 3. `bun install` if a dep is missing. | | `kata-manifest.json` clean | REQUIRED | Anti-duplication source of truth (Critical Rule #12). `bun run kata:manifest:check` clean before proposing components/ATCs; `bun run kata:manifest` if stale. | | Active env + test-user creds | REQUIRED | Authored tests run live against `<<ACTIVE_ENV>>`. Env reachable + `.env` creds for the env (per role if multi-role). | | Playwright browsers | REQUIRED | `bunx playwright` resolves + chromium installed (`bun run pw:install`). | | OpenAPI MCP (schema read-only) + `api/schemas/` synced | SCOPE — API/integration tests; needed at **Phase 1 Plan** too | Phase 1 explores endpoints (via the `openapi` MCP, schema-read-only) to design ATCs + classify test-data — plan-time, not just run-time. Api components consume OpenAPI-derived types (`api/schemas/`; refresh `bun run api:sync`); authenticated test-code calls use the Playwright API fixture (`.auth/api-state.json` from `bun run api:login`) — no `API_TOKEN`/MCP injection, no restart. | | DBHub MCP | SCOPE — data setup/validation; needed at **Phase 1 Plan** too | Phase 1 explores the schema (via the `dbhub` MCP) to design data fixtures (Discover / Modify / Generate) — plan-time, not just run-time. `dbhub` answers a schema probe; `DBHUB_*` in `.env`. Unset → fill `.env` + RESTART. | | Issue-tracker (`[ISSUE_TRACKER_TOOL]`) | SCOPE — ticket/regression-driven | ATP + AC reads via `bun run jira:sync-issues`; TMS modality for the ATP source. Pure module-driven from an existing spec may not need it. | Surfaces (UI vs API vs both) follow the chosen planning scope + the ATCs Phase 1 designs — NEVER a user question (reference §5). After the gate clears (generic baseline + the surface tools the scope needs GREEN), continue to Phase 0 below. --- ## Phase 0 — Resume check (MANDATORY, inline) Before picking the planning scope, run the session resume contract from `agentic-qa-core/references/session-management.md` §4: 1. Determine the prospective `<scope>` from the invocation context (ticket key, regression-driven TC, or module slug — see "Pick the planning scope first" below). 2. Check `.session/test-automation/<scope>/progress.md`. 3. If it does NOT exist → proceed to scope picker + Phase 1. 4. If it DOES exist: - Read `plan.md` (thin index) + the tail of `progress.md`. - Read the cited canonical `spec.md` / `automation-plan.md` / `atc/*.md` under `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/` for the domain content. - Surface to the user: last completed phase (Plan / Code / Review) + next phase + open Review findings if any. - Offer **resume / restart / abort**. On `restart`, archive to `.session/.archive/<YYYY-MM-DD>-test-automation-<scope>-aborted/` before proceeding. Phase 0 is inline (no subagent). It runs in <1 minute on a cold cache. --- ## Pick the planning scope first Every automation session starts by choosing one of three planning scopes. Pick once, then follow the Plan → Code → Review pipeline. | Scope | Input | Output | Use when | |-------|-------|--------|----------| | **Module-driven** (Macro) | A module name + list of candidate TCs | One module spec + N ATC specs | Batch-automating an entire module (10+ tests). First pass on a new area. | | **Ticket-driven** (Medium) | A single ticket/story ID with scenarios | One implementation plan for that ticket | Automating one user story end-to-end. Default for sprint work. | | **Regression-driven** (Micro) | One specific TC (often after a bug fix) | One ATC implementation plan | Adding a single regression test after a fix. Smallest unit of work. | When in doubt, ask the user which scope. Never assume "module" just because multiple TC IDs appear in the briefing. **These scopes consume the `Candidate` verdicts from `/test-documentation`** (Stage 4) — only `Candidate` TCs reach automation; `Manual` / `Deferred` are terminal. The mapping from that skill's 4 documentation scopes: `Module (Macro) ← module-driven`, `Ticket (Medium) ← ticket-driven`, `Regression-driven (Micro) ← bug-driven`. Candidates from an `ad-hoc / exploratory` documentation session enter under whichever fits — a module batch, or regression-driven for a single TC. --- ## Batch mode (fleet) — optional second executor A batch (module-driven, or several ticket-driven scopes queued together) runs **sequentially in one session by default**: one Work Package at a time, Plan → Code → Review each. That is this skill's behaviour and it does not change. A **batch fleet** — several persistent sessions working different Work Packages at once, coordinated by a conductor — is opt-in. Enter it only when the user asks for it, or when the batch holds 3+ Work Packages that touch **disjoint modules**. Everything below is a scoping rule; the launch and lifecycle transport lives in `orca-orchestration/SKILL.md` (`[ORCHESTRATION_TOOL]`), never here. - **Work Package (WP) = one delivery unit = one `test-specs/<ID>/` spec** (`spec.md` + `automation-plan.md` + `atc/*.md` under the Epic's `test-specs/` tree). Sprint origin: the `Candidate` TCs of one Story. Discovery origin: a Tech Story. A batch is a list of WPs, never a list of files. - **Partition by module, not by ticket.** One worker owns every WP that touches a module's components; WPs sharing a module run **sequentially inside that worker**. NEVER two workers on the same module — same-module WPs share Pages / Apis / fixtures and collide in the files with the least merge tolerance. - **One worktree per worker.** This skill writes code, and two sessions in one checkout contend on the git index even when their files are disjoint. Branch + PR per `git_strategy` (`sdet` = one trunk, see `.agents/skills/git-flow-master/references/sdet-integration-trunk.md`). - **The conductor regenerates `kata-manifest.json` per integration** (`bun run kata:manifest`). It is generated output — never hand-merged, never resolved as a text conflict. - **The batch runs with or without an orchestration binary.** The conductor always writes the launch file (one self-contained line per worker); with the binary those exact lines are launched for it, without it the human pastes them. Same payload either way, and nothing about the absence is reported to the user. Full protocol — partition algorithm, collision table, conductor-only operations, integration order, per-worker brief: `references/batch-fleet.md`. --- ## Workflow — Plan → Code → Review ``` Phase 1: Plan -> Phase 2: Code -> Phase 3: Review (spec / plan) (component + test file) (KATA compliance) | | | .context/PBI/epics/ tests/components/** Review checklist EPIC-<KEY>-<slug>/ tests/e2e/** or (pass/fail) test-specs/<scope>/ tests/integration/** spec.md automation-plan.md atc/*.md Register in fixture ``` Each phase has a gate. Do not start Code before the Plan is written and approved. Do not close out a ticket until Review passes. ### Phase 1 — Plan **MUST-load before any planning**: `kata-manifest.json` (root). It lists every Component and every ATC currently in the codebase. Use it to identify reuse, avoid duplicate `Page`/`Api` classes, and avoid minting an `@atc('PROJ-XXX')` ID that is already taken. This is enforced by Critical Rule #12 in `AGENTS.md` and by the husky pre-commit gate. **Pre-flight checklist** (anti-duplication — run before writing the plan): - Load `kata-manifest.json`. Cross-check every proposed TC ID against `components.api[].atcs[].id` and `components.ui[].atcs[].id`. If a match exists, the TC is already automated — re-scope or reuse. - Cross-check every proposed Component name against `components.api[].name` and `components.ui[].name`. If a match exists, extend the existing class — do not create a new one. - If reuse opportunity exists (same flow already covered by a Steps method or ATC), adapt the plan to extend rather than rebuild. Write the canonical domain plan file(s) for the chosen scope under the Epic's `test-specs/` tree, `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/` (`spec.md`, `automation-plan.md`, `atc/*.md`). These are NON-Jira hand-authored files (committed to git). The automation `automation-plan.md` is distinct from the Story-folder dev `implementation-plan.md` (Jira-synced, read-only). The plan answers: - Which scenarios from the ticket become tests, which become ATCs, which are shared preconditions (Steps)? - Which components already exist (`tests/components/api/*Api.ts`, `tests/components/ui/*Page.ts`) and which need to be created? - What test data is required? Classify by Discover / Modify / Generate (never assume data exists in staging). - Which fixture will the test use -- `{api}`, `{ui}`, or `{test}`? Do any shared preconditions call for a Steps class (`tests/components/steps/*Steps.ts`), instantiated directly in the test? - Which ATC IDs (from the TMS) map to which component methods? **Also write the session index `plan.md`** at `.session/test-automation/<scope>/plan.md` per `agentic-qa-core/references/session-management.md` §6. This is a THIN INDEX — Goal, Inputs (cites the canonical artifacts above), Approach, Phase breakdown table, Risks, Verification checklist, Cross-references. It does NOT duplicate the domain content; it points to it. Use the dispatch defined in §Subagent Dispatch Strategy: **Single**. Full briefing in `references/planning-playbook.md` §Plan dispatch. Present the plan to the user. Wait for approval before coding. After approval, the orchestrator appends `## Phase 1 — Plan — <ts>` with `status: completed`, `artifacts_touched: [list of domain + session files]`, `next: Phase 2 — Code` to `.session/test-automation/<scope>/progress.md`. ### Phase 2 — Code Use the dispatch defined in §Subagent Dispatch Strategy: **Sequential** (one subagent per scope unit). The subagent loads `references/e2e-patterns.md` and `references/api-patterns.md` per scope. **Skills to load in every Code subagent (mandatory)**: `/playwright-best-practices` (community, project-installed) for upstream Playwright/TypeScript patterns — flaky-test fixes, POM vs fixtures, axe-core, auth/OAuth, fixtures lifecycle, perf budgets, i18n, component testing. Load **alongside** `/test-automation` (this skill, project-authored) — the two are complementary: KATA-specific rules (ATC identity, inline locators, fixture selection) come from here; generic Playwright craft comes from `/playwright-best-practices`. Add `/playwright-cli` only when the subagent also needs to drive a real browser session (snapshot/trace/record) during code-time exploration. **Open the TC lifecycle first.** Before the first line of code, move every in-scope TC out of `{{jira.status.test_case.candidate}}`: `[ISSUE_TRACKER_TOOL] Transition: {{jira.transition.test_case.start_automation}}` → `{{jira.status.test_case.in_automation}}`. A TC left at `candidate` while its code is being written tells the team nobody picked it up. Unmapped slug → `agentic-qa-core/references/artifact-lifecycle.md` §4 fallback (ask, never skip silently). Implement in this order: 1. **Types** at top of component file (payloads, responses, domain DTOs). 2. **Component class** extending `ApiBase` or `UiBase`. Helpers first (no decorator), ATCs second (`@atc('TICKET-ID')`). 3. **Register** the component in `tests/components/ApiFixture.ts` or `UiFixture.ts` as appropriate. Steps classes (`tests/components/steps/*Steps.ts`) are NOT registered in any fixture — tests and setup files instantiate them directly (see `ExampleSteps.ts`). 4. **Test file** under `tests/e2e/{module}/` or `tests/integration/{module}/`, using the correct fixture. 5. **Run + verify**, in this exact order -- do not skip steps: ```bash bun run test <path/to/new.test.ts> # does it pass? bun run types:check # tsc --noEmit, no errors bun run lint:check # ESLint, no errors ``` If any step fails, fix before moving to Review. 6. **Signal it in the TMS** — per TC, both signals, neither optional: - **Link**: bind the automated test to the manual `Test` it automates via the `test_automation` link type (`{{jira.link_types.test_automation}}`, outward `automation test for` — slug-resolved, never a literal; catalog: `agentic-qa-core/references/traceability-linking.md` §3). - **Labels**: ADD `automated` on the `Test` and REMOVE `automation-candidate` — the two are mutually exclusive (`test-documentation/SKILL.md` §Labels). Flip them at the point the TC actually reaches **AUTOMATED** (see §Git & TMS handoff: the `merged` transition after the suite PR lands on `main` with CI green), not on a merge into the integration trunk. ``` [ISSUE_TRACKER_TOOL] Link work items: {AUTOMATED_TEST_KEY} -> {TC_KEY} type: {{jira.link_types.test_automation}} [ISSUE_TRACKER_TOOL] Update work item: {TC_KEY} labels: + automated - automation-candidate ``` **Progress checkpoint**: after each Code subagent returns (per scope unit — one per TC for module-driven, one total for ticket-driven), the orchestrator appends a phase entry to `.session/test-automation/<scope>/progress.md` per `agentic-qa-core/references/session-management.md` §7. For module-driven scope, mid-batch resume reads the entries and skips already-coded ATCs. #### AI-readable verification (optional, recommended) For the test you just wrote, run Allure 3 in **agent mode** to get a markdown report you can read directly without parsing HTML: ```bash bun allure:agent # runs `bunx allure agent -- bun test` ``` Allure 3 lives as a devDep — `bunx allure` resolves to the local `node_modules/.bin/allure`, no global install required. Use this when: - The Code subagent needs to confirm the test actually exercised the expected ATC (the markdown summary lists each `@atc('TICKET-ID')` block + its status). - You want a quick scope check before opening Phase 3 — Review. - You are debugging a fixture/locator mismatch and want a structured failure read instead of a raw stack. For human review of the same run, switch to: ```bash bun allure:run # bunx allure run -- bun test (full HTML report) bun allure:open # serve the last generated report locally ``` See `regression-testing` for CI / suite-level reporting (`bun allure:generate`, `bun allure:watch`). ### Phase 3 — Review Use the dispatch defined in §Subagent Dispatch Strategy: **Parallel** (3 simultaneous Verifiers). Full briefings in `references/review-checklists.md` §Parallel verification dispatch. Run the review checklist on the new/modified files. Treat every failed item as a blocker. A clean review is the merge gate. See `references/review-checklists.md` for the full lists (E2E and API have overlapping but distinct checklists). **Required separate verifier** — before merge, run `/pr-review-lead` or `/judgment-day` against the diff in a clean context (a fresh session/subagent with no memory of how the code was written). This is not opt-in and not limited to high-risk changes: Automation is the only stage whose autonomy reaches 3, and per `agentic-qa-core/references/stage-gates.md` it is the only stage with a mandatory separate verifier — more rope on the way in is paid for with a harder check on the way out. `/judgment-day` runs two blind judges in parallel against the diff and only approves when both agree (see `.agents/skills/judgment-day/SKILL.md`); `/pr-review-lead` runs a QA-lead-style review grounded in KATA doctrine. Pick whichever fits the change; skipping this step is a Review DoD failure, not a shortcut. **Light stage verifier** (closes the Automation stage) — run the eight-line template in `agentic-qa-core/references/artifact-lifecycle.md` §5. Stage-specific lines: ``` [ ] Every in-scope TC at {{jira.status.test_case.in_automation}} or beyond (start_automation fired) [ ] TCs bound to their automated test via the {{jira.link_types.test_automation}} link [ ] Labels flipped only at the status they belong to (+automated / -automation-candidate at `merged`, never at trunk merge) [ ] No TC left at {{jira.status.test_case.candidate}} with code already written for it [ ] Any unmapped slug went through the §4 fallback (asked), never a silent skip ``` **Progress checkpoint + Archive**: after Phase 3 returns ACCEPT (all 3 Verifiers exit 0), the orchestrator appends `## Phase 3 — Review — <ts>` with `status: completed`, `next: stop` to `.session/test-automation/<scope>/progress.md`, then runs Archive per `agentic-qa-core/references/session-management.md` §8: moves `.session/test-automation/<scope>/` to `.session/.archive/<YYYY-MM-DD>-test-automation-<scope>/` (two-file dir preserved) and calls `mem_session_summary` including the archive path. On REJECT, archive does NOT run — the working directory stays for debug. #### Git & TMS handoff (sdet integration-trunk suites) This skill stops at a clean local review. It does **not** create branches, push, or open PRs — that is `/git-flow-master`'s job. When the repo's git strategy is `sdet` (the standing mode for chained test-automation suites), each ticket flows through the per-ticket loop in `.agents/skills/git-flow-master/references/sdet-integration-trunk.md`: - The Phase 3 ACCEPT gate (3 Verifiers green: `test` / `types:check` / `lint:check`) is the skill's **local validation gate**. Under `sdet` it must pass on **both** the `local` and `staging` environments before push — re-run the suite against each (`active_env` per `.agents/project.yaml`). The Verifiers are local-only; Sanity CI on the branch is owned by `/git-flow-master` + `/regression-testing`, never by this skill. - After ACCEPT, surface the explicit handoff — _"Local gate green. Ready for `/git-flow-master`: cut `test/{KEY}-{slug}` from the integration trunk, push, Sanity-CI, PR into the trunk, merge `--no-ff`."_ Do not auto-invoke git operations. - **Append the Git Ledger line** to the suite's `progress.md` after each branch action (orchestrator-written, append-only) so a resuming session knows how the trunk was left: trunk name + SHA, last ticket merged, pending tickets, sync-gate / final-PR state. Schema in `../agentic-qa-core/references/session-management.md` §7 "The Git Ledger"; what-to-write detail in `.agents/skills/git-flow-master/references/sdet-integration-trunk.md` §Resume. - **TC lifecycle anchors to the ticket-branch PR, not the final `trunk → main` PR.** The full ladder this skill owns (canon: `agentic-qa-core/references/artifact-lifecycle.md` §1, Test row): | Moment | Transition | Status after | |---|---|---| | Phase 2 — Code opens | `{{jira.transition.test_case.start_automation}}` | `{{jira.status.test_case.in_automation}}` | | ticket PR opens into the trunk | `{{jira.transition.test_case.create_pr}}` | `{{jira.status.test_case.pull_request}}` | | final suite PR merges to `main`, CI green there | `{{jira.transition.test_case.merged}}` | `{{jira.status.test_case.automated}}` | Merging into the trunk is NOT `automated`. Execute transitions via `/test-documentation` + `[ISSUE_TRACKER_TOOL]`; resolve every slug through `.agents/jira-workflows.json`, and on an unmapped slug run the `artifact-lifecycle.md` §4 fallback instead of skipping. --- ## Fixture selection (inline — load-bearing every invocation) The fixture you pick determines whether a browser opens. Wrong fixture = slow API tests or missing UI context. | Test type | Fixture | Browser opens? | Use when | |-----------|---------|----------------|----------| | API only (integration) | `{ api }` | No (lazy) | Pure API testing. No UI needed. | | UI only | `{ ui }` | Yes | UI-focused testing. No backend setup via API. | | Hybrid (UI + API setup) | `{ test }` | Yes | Setup data via API, drive flow via UI, verify via API. | Rules: - Only three fixtures exist: `{ api }`, `{ ui }`, `{ test }` (see `tests/components/TestFixture.ts`). Reusable precondition chains (3+ ATCs repeated across 3+ files) go in a Steps class under `tests/components/steps/` — instantiated directly in the test, never exposed as a fixture. - Integration tests (`tests/integration/**`) almost always use `{ api }`. - E2E tests (`tests/e2e/**`) use `{ ui }` if no API setup needed, otherwise `{ test }`. - Never request `{ ui }` for a test that never interacts with the UI -- it opens a browser for nothing. --- ## Gotchas (inline — the most common rejection reasons) 1. **ATC = complete test case, not a single click.** `clickLoginButton()` is not an ATC. `loginWithValidCredentials(credentials)` is. If the method is one-line-wrapping `page.click()`, delete it. 2. **TC Identity Rule: Precondition + Action = 1 TC.** All expected results from the same precondition and same action are assertions of the same TC, not separate TCs. Do not split a TC across panels, endpoints, or UI sections. 3. **Equivalence Partitioning.** Same expected output = one parameterized ATC. Three ATCs all returning HTTP 401 for invalid login are wrong -- merge into one `loginWithInvalidCredentials(payload)`. 4. **ATCs do not call ATCs.** ATCs are atomic. For reusable chains, use the Steps module (`tests/components/steps/*Steps.ts`). Steps are NOT decorated with `@atc`. 5. **Locators inline.** No `locators/*.ts` files. Put the selector in the ATC. If the same locator is used in 2+ ATCs of the same component, extract it to a `private readonly` arrow function in the class -- not to a separate file. 6. **Helpers vs ATCs.** A read-only GET is a helper (no `@atc`, optionally `@step`). An action that changes state is an ATC (`@atc('TICKET-ID')`). A GET inside an ATC that verifies the action succeeded is fine -- but the GET alone is not an ATC. 7. **Fixed assertions go inside ATCs.** Status code, required fields, URL redirect checks. Test-level assertions (flow outcomes) go in the test file. 8. **Max 2 positional parameters.** 3+ parameters must use an object parameter (`fn(args: Args)`). Named object parameters beat positional lists for maintainability and autocomplete. 9. **Import aliases are mandatory.** `@api/`, `@ui/`, `@utils/`, `@variables`, `@TestContext`, `@schemas/`. No relative imports (`../../../`). Lint will reject them. 10. **No hardcoded waits.** Never `page.waitForTimeout(3000)`. Wait for a specific condition: `waitForSelector`, `waitForResponse`, `waitForLoadState('networkidle')`, or `data-loaded="true"` attributes. 11. **No retries by default.** `retries: 0` in `playwright.config.ts`. If a test passes on retry, it is flaky, not passing. Investigate. 12. **Credentials from `.env`, never hardcoded.** `LOCAL_USER_EMAIL` / `STAGING_USER_EMAIL` and their passwords. Read via `config.testUser` from `@variables`. 13. **Each test generates its own data.** No shared state between tests. Use `TestContext.generateUserData()` or faker helpers for unique values. 14. **Ticket ID prefix in every `test()`.** Format: `test('TICKET-ID: should {behavior} when {condition}', ...)`. The `describe` block may also include the ticket ID when the file is tied to a single ticket. 15. **One component per file, one file per feature.** Components follow `{Resource}Api.ts` or `{Page}Page.ts`. Test files follow `{verb}{Feature}.test.ts` (e.g., `applyDiscount.test.ts`, never `discount.test.ts`). 16. **Don't propose components or ATCs without consulting the manifest.** `kata-manifest.json` is the registry. Skipping it produces (a) duplicate Pages — proposing `LoginPage` when `LoginPage.ts` already exists; (b) duplicate ATC IDs — minting `@atc('PROJ-90')` twice; (c) missed reuse — creating `getBookingById` when `BookingsApi.getById` already does it. Always start the Plan phase by loading the manifest. The husky pre-commit gate enforces freshness; Critical Rule #12 in `AGENTS.md` enforces consultation. 17. **Cross-cutting test-architecture decisions become ADRs, not plan-buried prose.** When Plan or Code reveals a decision that is architectural AND hard to reverse — a fixture lifecycle reused across 3+ ATCs or 2+ tickets, a test-data-isolation contract, an auth-in-tests change, a flake-retry-policy shift, a Page-Object-vs-Screenplay move — promote it from `planning-playbook.md` §2 "Architecture Decisions" to a standalone `.context/ADR/ADR-NNNN-<slug>.md` and leave a `See ADR-NNNN` backlink. Ticket-local choices stay in the plan. ADRs are append-only: supersede, never rewrite. See `agentic-qa-core/references/adr-doctrine.md`. 18. **Session-footer contract (mandatory at close).** The final phase is not done until the two chat-facing blocks from `../agentic-qa-core/references/session-footer-contract.md` are printed: (1) consolidated screenshot list — repo-relative paths, verified on disk, bug annotations first — plus in-flow surfacing of every capture's path the instant it lands; (2) Session Footer listing skills/MCPs/CLIs actually used + testing levels touched, with explicit "none" entries for expected-but-untouched levels. Framing for this skill: authoring. Multi-subagent sessions: each stage report carries the five footer fields (`skills_loaded`, `mcps_used`, `clis_used`, `testing_levels_touched`, `screenshots_captured`); the orchestrator compiles the footer ONCE at close. Chat only — never in a Jira comment or ATR body. --- ## Minimal templates (inline — small, load-bearing) ### KATA component signatures ```typescript // API component — Layer 3 export class UsersApi extends ApiBase { constructor(options: TestContextOptions) { super(options); } // Helper (read-only) @step async getUserById(id: string): Promise<[APIResponse, UserResponse]> { return this.apiGET<UserResponse>(`/users/${id}`); } // ATC (state-changing) @atc('TICKET-ID') async createUserSuccessfully(payload: UserPayload): Promise<[APIResponse, UserResponse, UserPayload]> { const [response, body, sent] = await this.apiPOST<UserResponse, UserPayload>('/users', payload); expect(response.status()).toBe(201); expect(body.id).toBeDefined(); return [response, body, sent]; } } ``` ```typescript // UI component — Layer 3 export class LoginPage extends UiBase { constructor(options: TestContextOptions) { super(options); } @atc('TICKET-ID') async loginWithValidCredentials(data: LoginData): Promise<void> { await this.page.goto('/login'); await this.page.locator('#email').fill(data.email); await this.page.locator('#password').fill(data.password); await this.page.locator('button[type="submit"]').click(); await expect(this.page).toHaveURL(/.*dashboard.*/); } } ``` ### Test file skeleton ```typescript import { test, expect } from '@TestFixture'; import usersData from '@data/fixtures/users.json'; test.describe('TICKET-ID: Validate discount codes', () => { test('TICKET-ID: should apply percentage discount when code is valid', async ({ api }) => { const order = await api.orders.createOrderSuccessfully(orderData); const totals = await api.orders.getTotals({ orderId: order.id }); expect(totals.finalAmount).toBe(totals.baseAmount - totals.discountAmount); }); }); ``` ### Fixture registration (excerpt) ```typescript // tests/components/UiFixture.ts export class UiFixture extends TestContext { readonly login: LoginPage; readonly checkout: CheckoutPage; constructor(options: TestContextOptions) { super(options); this.login = new LoginPage(options); this.checkout = new CheckoutPage(options); } } ``` --- ## Quality gates (must pass before merge) | Gate | Command | Must be | |------|---------|---------| | Tests pass | `bun run test {path}` | All green, zero retries used | | Types | `bun run types:check` | No errors | | Lint | `bun run lint:check` | No errors | | Fixture registered | visual | Component is in `ApiFixture` / `UiFixture` (Steps classes are instantiated directly, never registered) | | ATC IDs linked | visual | Every `@atc('X')` matches a real TMS test case ID | | Naming | visual | Files PascalCase for components, camelCase verb for test files | | Session footer | chat | Session footer + consolidated screenshot list printed in chat per session-footer-contract (never in a Jira comment) | --- ## Anti-patterns — NEVER do these **T1.** NEVER auto-generate tests for TCs that `/test-documentation` flagged as Deferred or Manual — only `Candidate` (`to_be_automated`) verdicts proceed to automation. Skipping the ROI verdict produces flaky, low-value suites. **T2.** NEVER skip the Plan phase. Even for a "simple" regression test, write `spec.md` / `automation-plan.md` (or the per-ATC plan under `.context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/`) BEFORE writing any test code. Plan → Code → Review is non-negotiable. **T3.** NEVER collapse the KATA layers (TestContext / ApiBase + UiBase / Domain Api+Page+Steps / Fixture). Full doctrine in `references/kata-architecture.md`. Tests that flatten layers are rejected at Review. **T4.** NEVER call one ATC from inside another ATC. ATCs are atomic mini-flows. Reusable chains live in the Steps module (`tests/components/steps/*Steps.ts`), which is NOT decorated with `@atc`. **T5.** NEVER use relative imports (`../../../`). This repo uses path aliases (`@api/`, `@ui/`, `@schemas/`, `@utils/`, `@TestContext`, `@variables`, `@TestFixture`). Lint rejects relative paths. **T6.** NEVER hardcode credentials. Read from `.env` via `config.testUser` from `@variables` (`LOCAL_USER_*` / `STAGING_USER_*`). Never inline a password, token, or API key in a spec, fixture, or test data file. **T7.** NEVER hardcode `customfield_NNNNN` in spec files, test data, or test config. Resolve Jira fields via `{{jira.<slug>}}` against `.agents/jira-fields.json` + `.agents/jira-required.yaml` so test code survives workspace rotations. **T8.** NEVER mix test code and product code in the same PR. Test PRs follow the `test/*` branch convention with title format `{type}({ISSUE-KEY}): {description}` — see `.agents/skills/git-flow-master/references/pr-test-automation.md`. Under the `sdet` strategy, adjacent non-test work never rides a `test/*` ticket branch either — it goes on a Plus Branch (`docs/*`/`chore/*`/`fix/*` → integration trunk). See `.agents/skills/git-flow-master/references/sdet-integration-trunk.md`. --- ## Which reference to read Not every invocation needs every reference. Load the specific file when the task matches. - **KATA architecture, fixture selection, ATC rules, Steps module mechanics** → `references/kata-architecture.md` - **TypeScript conventions (params, imports, types, errors, DRY by layer)** → `references/typescript-patterns.md` - **Naming, tagging, folder structure, anti-patterns, quality gates** → `references/automation-standards.md` - **Writing a Playwright `Page` component, locator strategy, data-testid rules, UI waits** → `references/e2e-patterns.md` - **Writing an `Api` component, OpenAPI type facades, HTTP helper usage, schema imports** → `references/api-patterns.md` - **Designing test data (Discover → Modify → Generate), fixtures JSON, faker** → `references/test-data-management.md` - **`@atc` / `@step` decorators, NDJSON results, TMS sync mechanics** → `references/atc-tracing.md` - **Writing the Plan (module / ticket / ATC scopes and templates)** → `references/planning-playbook.md` - **Running a batch across several parallel sessions (partition by module, conductor duties, integration order)** → `references/batch-fleet.md` - **Running the review checklist (E2E or API)** → `references/review-checklists.md` - **Configuring Playwright, CI integration, projects, sharding** → `references/ci-integration.md` - **Session resume contract, plan.md/progress.md schemas, archive policy, Engram per-phase checkpoint** → `../agentic-qa-core/references/session-management.md` (Phase 0 + Phase 1 + Archive of this skill) Tool resolution: use `[AUTOMATION_TOOL]` for browser work (Playwright CLI or MCP — load `/playwright-cli` when available), `[API_TOOL]` for OpenAPI exploration, `[DB_TOOL]` for verifying test data in the database, `[TMS_TOOL]` for TMS sync (load `/xray-cli` when available), `[ISSUE_TRACKER_TOOL]` for ticket work. Split the issue-tracker access by operation: **detailed reads** of a Story (ACs, ATP, dev implementation-plan, custom fields) → `bun run jira:sync-issues get <KEY> --include-comments` (or `jql "<query>"`) then read the synced `.md` — NEVER `acli workitem view` for custom fields; **writes** (comment automated-test status back to the Story, transitions) → `/acli`; **trivial summary/status/key-list lookups** → `/acli` `workitem view`/`search` is fine. See `agentic-qa-core/references/acli-integration.md` §"Reads vs writes". Resolve tags via the project's AGENTS.md Tool Resolution table. --- ## Quick reference ```bash # Planning outputs (hand-authored, NON-Jira; Epic-level test-specs/) # .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/spec.md # .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/automation-plan.md # .context/PBI/epics/EPIC-<KEY>-<slug>/test-specs/<scope>/atc/*.md # (the Story-folder implementation-plan.md is the Jira-synced DEV plan — read-only input, not written here) # Code locations # tests/components/api/{Resource}Api.ts # tests/components/ui/{Page}Page.ts # tests/components/steps/{Domain}Steps.ts # tests/integration/{module}/{verbFeature}.test.ts # tests/e2e/{module}/{verbFeature}.test.ts # Local validation loop bun run test <path> bun run types:check bun run lint:check bun run kata:manifest # regenerate registry if components/ATCs changed git add kata-manifest.json # stage so the freshness gate passes bun run kata:manifest:check # confirm gate would pass (husky runs this on commit) # Env + TMS sync cp .env.example .env # populate test credentials bun run api:sync # regenerate OpenAPI schema types ```
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.