framework-development
Framework evolution mode — evolves the QA boilerplate itself (KATA, fixtures, cli/, scripts/, api/schemas/ pipeline, package.json deps). Self-contained Plan → Code → Verify → Archive pipeline; runs under the `gentle-ai install --preset minimal` install (no SDD-* skills required).
Install
npx skills add https://github.com/upex-galaxy/agentic-qa-boilerplate/tree/main/.agents/skills/framework-development
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install upex-galaxy-agentic-qa-boilerplate@llmmart
git clone https://github.com/upex-galaxy/agentic-qa-boilerplate.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole upex-galaxy/agentic-qa-boilerplate collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Framework Development — Evolve the QA Boilerplate
Gateway skill for changes to the framework itself: KATA layers, fixtures, installer, OpenAPI pipeline, scripts, doctrine docs. Per-ticket QA work, test specs, and TMS documentation are owned by other workflow skills (/sprint-testing, /test-documentation, /test-automation, /regression-testing) and MUST NOT trigger this skill.
The skill exists because framework-surface changes — new fixture, new layer helper, installer rewrite, manifest extractor — deserve a planning gate before code. Per-ticket test writing already has its own gate in /test-automation Plan → Code → Review; this skill is its architectural-surface counterpart.
Compact Rules
- DO NOT: use this skill for per-ticket work — test writing is
/test-automation, manual QA is/sprint-testing, TMS docs are/test-documentation, suite runs are/regression-testing. This skill governs the architectural surface only. - DO: clear the readiness preflight, then run the Phase 0 path self-check against
references/kata-invariants.md§10 before dispatching anything. A FORBIDDEN path aborts and redirects to the skill named in the row; a path in neither table is ASKED about, never assumed. - WHEN one change spans both ALLOWED and FORBIDDEN paths: split it. This skill changes the base;
/test-automationmigrates the consuming specs in a follow-up. - DO: run Plan → Code → Verify → Archive in order for every non-trivial framework change. The pipeline IS the gate; "it's a quick refactor" is not an exemption.
- DO NOT: edit
tests/components/from a framework-development session — those L2/L3 KATA components are per-ticket surface. - DO NOT: collapse the KATA layers (TestContext / Base / Domain / Fixture) under a simplicity argument. They are framework architecture, not speculative abstraction.
- DO NOT: add a new fixture API without updating the matching fixture file AND
kata-manifest.jsonAND citing at least one existing test that consumes it. Orphan fixtures rot, and the manifest is the anti-duplication gate. - DO NOT: bump a major version of Playwright / Bun / TypeScript without a regression run on a representative E2E suite — lockstep upgrades hide breaks in fixture lifecycle, locator engines, and type emit.
- DO NOT: refactor
cli/install.tswithout exercising the full install flow on a clean clone. Verification on an already-installed repo proves nothing, and the installer is the one surface where a bug ships silently to every new user. - WHEN the chosen approach reshapes test architecture (KATA layers, a fixture API, the runner, the isolation/parallelization model, the OpenAPI/type pipeline) AND is hard to reverse: record an ADR under
.context/ADR/after plan approval and before coding, draftedProposedfor the human to accept. ADRs are append-only — supersede, never rewrite. - DO: verify with all four checks (test, types, lint, skills) and treat any non-zero exit as REJECT — present retry / skip-and-document / abort, never auto-fix.
- DO NOT: let a subagent write
progress.md; it is orchestrator-only. Code subagents return one-line summaries per task, and the orchestrator does not read their diffs. - DO: archive the session directory only after all four verifiers pass. On REJECT it stays in place so the run can be debugged or resumed.
Read full SKILL.md when: writing the plan artifact, batching Code-phase tasks, resuming an interrupted session, or reading the ALLOWED/FORBIDDEN path tables themselves.
Inputs
Canonical reading order for any AI starting cold on a framework-development workflow. Read in order; stop earlier when the change is small enough that later inputs add no signal.
kata-manifest.json— Component + ATC registry (source of truth per Critical Rule #12). Establishes what already exists before any new fixture API, Page, Api, Steps module, or ATC ID is proposed..agents/skills/test-automation/references/kata-architecture.md+.agents/skills/test-automation/references/typescript-patterns.md— KATA layer flow (TestContext → ApiBase / UiBase → YourApi / YourPage → TestFixture), ATC identity rules, fixture-selection contract, import-alias conventions.tests/components/— current Api / Page / Steps shape; required reading when touching any L2 / L3 surface or adding a fixture consumed by these components.cli/install.ts— installer flow; required reading when evolving the installer, adding install steps, or modifying boilerplate scaffold behavior.scripts/sync-openapi.ts+api/schemas/— OpenAPI-derived TypeScript types pipeline; required reading when touching the API contract pipeline, schema generation, or any consumer of generated facades.package.json+bun.lockb— dep landscape; required reading before bumping Playwright / Bun / TypeScript / fixture-runtime versions or adding/removing scripts.
Subagent Dispatch Strategy
Orchestration & Session contracts: this skill follows
agentic-qa-core/references/orchestration-doctrine.md(mandatory subagent dispatch — main thread is command center) ANDagentic-qa-core/references/session-management.md(Phase 0 resume check, plan-first persistence at.session/<skill-slug>/<scope>/, archive on completion). Phase 0 (resume check) and Phase 1 (plan write) are NOT optional.
This skill is compliant with the doctrine in AGENTS.md §"Orchestration Mode (Subagent Strategy)" and the session contract in .agents/skills/agentic-qa-core/references/session-management.md. Every dispatch follows the 7-component briefing format defined in .agents/skills/agentic-qa-core/references/briefing-template.md, and the pattern selected per phase matches the decision guide in .agents/skills/agentic-qa-core/references/dispatch-patterns.md. The four phases — Plan, Code, Verify, Archive — mirror the shape of /test-automation (Plan → Code → Review) extended with an inline Archive step. Phase 0 stays inline because the path self-check + session resume check are short orchestrator decisions that do not benefit from a fresh-context subagent.
Session scope: <change-name> (kebab-case, user-provided at session start). Session state lives at .session/framework-development/<change-name>/{plan.md, progress.md} per agentic-qa-core/references/session-management.md §9.
| Phase | Pattern | Subagent role |
|---|---|---|
| Phase 0 — Path self-check + session resume check | inline | orchestrator only; no subagent. Lists target paths against references/kata-invariants.md §10; aborts on FORBIDDEN. Also checks .session/framework-development/<change-name>/ for prior plan/progress and offers resume per agentic-qa-core/references/session-management.md §4 |
Phase 1 — Plan (plan.md) |
Single | one Plan subagent writes .session/framework-development/<change-name>/plan.md per agentic-qa-core/references/session-management.md §6 schema; collapses prior explore + propose + spec + design + tasks into one artifact |
| Phase 2 — Code (per task batch) | Sequential | one Code subagent per task batch from the plan; orchestrator appends to progress.md per agentic-qa-core/references/session-management.md §7 schema; on verification failure: STOP, no auto-fix |
Phase 3 — Verify — bun run test |
Parallel (sub-stage) | one Verifier subagent runs the test suite |
Phase 3 — Verify — bun run types:check |
Parallel (sub-stage) | one Verifier subagent runs typecheck |
Phase 3 — Verify — bun run lint:check |
Parallel (sub-stage) | one Verifier subagent runs ESLint |
Phase 3 — Verify — bun run skills:check |
Parallel (sub-stage) | one Verifier subagent runs the skill-registry lint (framework changes can affect .agents/skills/, AGENTS.md, cli/install.ts) |
| Phase 3 — Aggregation + accept/reject decision | inline | orchestrator reads the 4 Verifier reports and decides; on any non-zero exit, presents retry / skip / abort |
| Phase 4 — Archive (move plan + progress) | inline | orchestrator only; moves .session/framework-development/<change-name>/ to .session/.archive/<YYYY-MM-DD>-framework-development-<change-name>/ (two-file dir preserved per agentic-qa-core/references/session-management.md §8); references in commit |
- Plan artifact location:
.session/framework-development/<change-name>/plan.md. The.session/tree is gitignored — the plan is local, not committed. Recovery on mid-run crash: the file persists; the orchestrator reads it back on the next session via Phase 0 resume check (seeagentic-qa-core/references/session-management.md§4). - Grace period for legacy path: prior versions wrote to
.scratch/framework-changes/<change-name>/{plan.md, apply-progress.md}. Phase 0 also checks the legacy path during the grace period — if found, the orchestrator offers to copy state to the new.session/...location before resuming. - Path guardrails injected per dispatch: every Plan and Code subagent briefing MUST include the line
KATA invariants and ALLOWED/FORBIDDEN paths: .agents/skills/framework-development/references/kata-invariants.md (read §10 before touching any file).Do NOT inline the path tables — the reference is authoritative. - On any subagent failure: STOP, return the failing report, do NOT auto-rerun. The orchestrator decides retry / skip / abort. See
.agents/skills/agentic-qa-core/references/orchestration-doctrine.md. - Strict TDD flag is set in Phase 1's
plan.mdunder §"Strict TDD flag". Default OFF. Flipped ON only when the user explicitly opted in. Code phase reads it from the plan; no separate cache needed.
Fleet seam (optional)
Phase 2 Code is sequential per task batch by default, and a change that fits one plan stays that way. A wave — several persistent worker sessions taking one task batch each, a conductor integrating them — is for a plan whose batches are genuinely independent (different files, no shared public API in flight). Offer it; never enter it silently.
- Topology: same checkout with file ownership per worker. Each worker's brief lists the exact files it owns and commits with explicit paths (never
git add -A), so two workers never stage each other's work. When two batches must touch the same module, they do not run in the same wave: serialize them, or give each worker its own worktree. - The conductor writes
launch.txtin.session/framework-development/<change-name>/— one self-contained line per worker — always, whether or not any orchestration transport exists on the machine. Launching, supervising and closing those sessions isorca-orchestration/SKILL.md([ORCHESTRATION_TOOL]): supervised launch is the native path, andlaunch.txtis the payload for the human-paste fallback when nothing can launch it. - Phase 0 and Phase 3 stay with the conductor. The path self-check runs once, over the union of every worker's paths, before the first launch; the four Phase 3 verifiers run once, on the integrated tree, after the last worker reports. A per-worker green gate is not a wave gate.
- Read before every edit. A worker's neighbour may have changed a shared file since the brief was written; Critical Rule #15 (no global discards) binds twice as hard when sessions share a tree.
- Silence rule: the absence of an orchestration transport is never named to the user and never appears in the preflight gate or the plan.
Readiness Preflight Gate (MANDATORY — runs before Phase 0)
Full doctrine:
agentic-qa-core/references/preflight-gate.md. Runs FIRST, before the path self-check and resume check. Two laws: (1) args-as-answers — the change name and touched paths are provided args; ask only the gaps. (2) probe, don't assume. Surface gaps + REDs as ONEAskUserQuestionchecklist; self-fix with approval + explanation; STOP on any blocking RED. This skill evolves the framework itself — it does NOT hit a live env, Jira, DB, or API — so its gate is a dev-toolchain readiness check that pairs with the Phase 0 path self-check. Generic baseline (the two laws, secret/restart handling, output contract) is inherited from the reference §3.1 — not repeated here; the env/creds half of the baseline is N/A for meta-work. Below is only this skill's specific capability delta.
| Capability | Need | Why here |
|---|---|---|
| Dev toolchain | REQUIRED | Phase 3 Verify runs bun run test / bun run types:check / bun run lint:check / bun run skills:check. All four must resolve at t=0. bun install if a dep is missing. |
kata-manifest.json clean |
REQUIRED | Source of truth (Critical Rule #12). Framework changes can invalidate it — bun run kata:manifest:check clean, bun run kata:manifest to regenerate. |
| Playwright browsers | SCOPE — touching fixtures / KATA bases / tests | Verify of a fixture or base-class change runs the suite, which needs chromium (bun run pw:install). |
/github-actions-docs + /playwright-best-practices |
OPTIONAL | Injected per dispatch when the change touches CI YAML or fixtures/tests (already noted in the briefing skeleton). |
Active env, test-user creds, OpenAPI/API_TOKEN, DBHub, issue-tracker, TMS and resend are N/A — framework evolution is meta-work on this repo. After the gate clears (all REQUIRED GREEN), continue to Phase 0 below.
Phase 0 — Path self-check + session resume check (mandatory, runs first)
Before invoking any subagent, the orchestrator MUST (a) list the files / directories the change will touch and verify each one against the ALLOWED / FORBIDDEN tables in references/kata-invariants.md §10, and (b) run the session resume check per agentic-qa-core/references/session-management.md §4. Skipping Phase 0 is the most common way framework changes leak into ticket-owned surface area OR lose mid-run state on interruption.
- Read
references/kata-invariants.md§10 (ALLOWED + FORBIDDEN paths). - Ask the user (or infer from the request): "Which paths will this change touch?"
- For each path, look it up in §10 ALLOWED → proceed. Or §10 FORBIDDEN → abort and redirect to the skill named in the row.
- If a path matches neither table, ASK the user explicitly — never assume.
- If a single change spans both ALLOWED and FORBIDDEN paths (e.g. "refactor
tests/components/ui/UiBase.tsAND update the e2e tests that consume it"), split the work: framework-development handles the base-class change;/test-automationhandles the test-spec migration in a follow-up. - Session resume check (per
agentic-qa-core/references/session-management.md§4): check.session/framework-development/<change-name>/progress.md. If it exists, readplan.md+ the tail ofprogress.md, surface the last completed phase + next planned phase + any blocking notes, and offer resume / restart / abort. Onrestart, archive the current directory to.session/.archive/<YYYY-MM-DD>-framework-development-<change-name>-aborted/before proceeding. - Legacy path check (grace period): also check
.scratch/framework-changes/<change-name>/for prior plan/progress under the old layout. If found, offer to migrate the state to the new.session/...location.
Phase 0 is one short inline decision — it does NOT write a file. If the change is approved, the decision is captured later in Phase 1's plan.md §Investigation.
Native Phase Orchestration
After Phase 0 passes, run the four-phase pipeline in dependency order. Each subagent dispatch follows the 7-component briefing format in agentic-qa-core/references/briefing-template.md. Briefing skeleton for Plan and Code phases (fill the <...> slots):
Goal: <one-sentence outcome scoped to this phase>
Context docs:
- .agents/skills/framework-development/references/kata-invariants.md
- .session/framework-development/<change-name>/plan.md (Code phase only)
- .session/framework-development/<change-name>/progress.md (Code phase, batches > 1; orchestrator-written, read-only for subagents)
- <relevant ALLOWED-path files the phase will read or touch>
Project Standards (auto-resolved):
<compact-rule blocks pulled from .agents/skills/REGISTRY.md per skill-resolver protocol>
Skills to load: <none by default; orchestrator injects /playwright-best-practices if fixtures/tests, /github-actions-docs if CI YAML>
Exact instructions:
1. Read kata-invariants.md fully. Verify §10 ALLOWED for every touched path. FORBIDDEN → STOP.
2. <phase-specific step — e.g. "write the plan", "implement task batch N">
3. Save the artifact at the engram topic_key framework/<change-name>/<phase>.
4. Return the executive summary inline.
Report format:
- status: ready | blocked | failed
- artifact: <absolute path or engram topic_key>
- next_recommended: <phase or "stop">
- risks: [<one-liner per risk>]
- skill_resolution: injected | fallback-inline
Rules:
- ALLOWED paths only (kata-invariants.md §10). FORBIDDEN → abort.
- Do NOT modify generated artifacts (api/openapi-types.ts, kata-manifest.json, reports/).
- If strict TDD is ON (read from plan §"Strict TDD flag"), every production-code task is preceded by a failing test in the same batch.
- On uncertainty, STOP and report — do not improvise on framework surface.
Phase order (each phase gates the next):
Phase 0 (inline) -> Phase 1 Plan (Single) -> Phase 2 Code (Sequential per batch) -> Phase 3 Verify (Parallel 4-way) -> Phase 4 Archive (inline)
Phase 1 — Plan
Dispatch: Single. The Plan subagent writes one consolidated artifact at .session/framework-development/<change-name>/plan.md covering: Goal, Investigation, Approach options, Chosen approach, Invariants touched, Public API delta, Task breakdown, Strict TDD flag, Risks, Verification checklist. One file, ten sections — not five separate documents. The seven base sections from agentic-qa-core/references/session-management.md §6 are mandatory; framework-development extends them with three skill-specific sections (Invariants touched, Public API delta, Strict TDD flag).
Present the plan to the user. Wait for approval before Phase 2.
ADR seeding (framework architecture). When the chosen approach reshapes the framework's test architecture — KATA layers, fixture APIs, the test runner, the isolation/parallelization model, or the OpenAPI/type pipeline — and the decision passes the two-gate test (architectural AND hard to reverse per agentic-qa-core/references/adr-doctrine.md §1), record a .context/ADR/ADR-NNNN-<slug>.md after the plan is approved and before Phase 2 coding. Framework evolution is meta-work: its decisions bind every test session that follows, so historicize them rather than leaving them in a one-off plan.md that gets archived. The plan's "Invariants touched" / "Public API delta" sections are the prime ADR candidates. Draft Proposed; the human accepts. Template + lifecycle: .context/ADR/README.md.
Phase 2 — Code
Dispatch: Sequential — one subagent per task batch. The orchestrator decides batching from the plan's task list; rule of thumb is 1 batch per 3-5 closely-coupled tasks, or 1 batch per task when the task touches a load-bearing file such as ApiBase.ts / UiBase.ts / TestContext.ts.
Each subagent:
- Reads
plan.mdand applies the tasks in its batch in plan order. - Runs the per-task verification command listed in the plan.
- Returns a one-line summary per task to the orchestrator.
- On verification failure: STOP, report, do not auto-fix.
The orchestrator never reads diffs from the Code subagent — only the summary. If the user wants to see actual changes, the orchestrator runs git diff inline after the batch returns. After each batch returns, the orchestrator appends a phase entry to .session/framework-development/<change-name>/progress.md per agentic-qa-core/references/session-management.md §7 (subagents never write to progress.md directly — that is an orchestrator-only file).
Phase 3 — Verify
Dispatch: Parallel — four Verifier subagents in the same <function_calls> block:
| Verifier | Command | Captures |
|---|---|---|
| V1 | bun run test |
exit code, summary |
| V2 | bun run types:check |
exit code, summary |
| V3 | bun run lint:check |
exit code, summary |
| V4 | bun run skills:check |
exit code, ERROR/WARN/INFO counts |
skills:check is included because framework changes routinely touch .agents/skills/framework-development/, agentic-qa-core/references/, AGENTS.md, and cli/install.ts — every one of those surfaces is read by scripts/lint-skills.ts and gated by 10 named checks (tier coherence, anti-leak, stale-path, duplicate-tier, etc.). The other three commands never see this surface; adding the fourth verifier costs one parallel slot and prevents an entire failure class.
After all four return, the orchestrator inline-aggregates:
- All four
exitCode == 0→ ACCEPT. Proceed to Phase 4. - Any
exitCode != 0→ REJECT. Present failing verifier(s) to the user. Options: retry the failing Phase 2 batch / skip-and-document / abort. Do NOT auto-fix.
Phase 4 — Archive
Dispatch: inline — no subagent. The orchestrator performs the archive flow from agentic-qa-core/references/session-management.md §8:
- Verifies the Verification checklist in
plan.mdpasses (all four Phase 3 verifiers returned exit 0). - Moves the entire working directory:
mv .session/framework-development/<change-name>/ .session/.archive/<YYYY-MM-DD>-framework-development-<change-name>/. Bothplan.mdandprogress.mdare preserved side by side (no concatenation) so future resume-replay stays possible. - Calls Engram
mem_session_summarywith the session template peragentic-qa-core/references/session-management.md§11. The summary MUST include the archive path somem_search "session framework-development <change-name>"resolves back to the artifacts. - Surfaces the archive path so
/git-flow-mastercan include it in the commit message body.
Archive is a "close-the-loop" step, not "ship-the-code". Code is shipped by /git-flow-master based on the diff that Phase 2 produced and Phase 3 verified. On Phase 3 REJECT, archive does NOT run — the working directory stays in place so the user can debug, resume, or abort.
Anti-patterns — NEVER do these
- F1. NEVER use
/framework-developmentfor per-ticket test writing — that surface is owned by/test-automation(Plan → Code → Review on KATA + Playwright + TypeScript). Framework-development governs the architectural surface only. - F2. NEVER collapse KATA layers (TestContext / Base / Domain / Fixture) under the pretext of simplification. The layers are framework architecture, not speculative abstraction. Critical Rule #12-adjacent: simplicity-first does NOT apply to KATA.
- F3. NEVER edit
tests/components/from a framework-development session — those are L2 / L3 KATA components owned by per-ticket work via/test-automation. If a base-class refactor forces a consumer migration, split the work: framework-development changes the base;/test-automationmigrates the specs in a follow-up. - F4. NEVER skip the Plan → Code → Verify → Archive pipeline for non-trivial framework changes. The pipeline IS the gate — bypassing it for "quick" refactors of
ApiBase.ts,UiBase.ts,TestContext.ts, fixtures, installer, or OpenAPI pipeline reliably produces undetected regressions. - F5. NEVER bump major versions of Playwright / Bun / TypeScript without a regression run on a representative E2E suite. Lockstep upgrades hide breaking changes in fixture lifecycle, locator engines, or type-emit behavior.
- F6. NEVER add a new fixture API without updating
tests/components/TestFixture.ts(or the matchingApiFixture.ts/UiFixture.ts) ANDkata-manifest.jsonAND citing at least one existing test that consumes it. Orphan fixtures rot — andkata-manifest.jsonis the anti-duplication gate (Critical Rule #12). - F7. NEVER refactor
cli/install.tswithout testing the full install flow on a clean clone. The installer is the only surface where a bug ships silently to every new user — verification on the developer's already-installed repo proves nothing. - F8. NEVER introduce a hard-to-reverse test-framework architectural decision (KATA-layer reshape, new fixture API, test-runner swap, isolation/parallelization model) without recording it as an ADR in
.context/ADR/. Framework evolution binds every later test session — a decision left only in an archivedplan.mdgets re-litigated or silently violated. DraftProposedbefore Phase 2; the human approves. ADRs are append-only: supersede, never rewrite. Seeagentic-qa-core/references/adr-doctrine.md.
Session close contract
Session-footer contract (mandatory at close). The final phase is not done until the two chat-facing blocks from ../agentic-qa-core/references/session-footer-contract.md are printed: (1) consolidated screenshot list — repo-relative paths, verified on disk, bug annotations first — plus in-flow surfacing of every capture's path the instant it lands; (2) Session Footer listing skills/MCPs/CLIs actually used + testing levels touched, with explicit "none" entries for expected-but-untouched levels. Framing for this skill: meta. Multi-subagent sessions: each stage report carries the five footer fields (skills_loaded, mcps_used, clis_used, testing_levels_touched, screenshots_captured); the orchestrator compiles the footer ONCE at close. Chat only — never in a Jira comment or ATR body.
References
references/kata-invariants.md— INVARIANT vs EXTENSIBLE rules for the 4 KATA layers, fixture selection, ATC identity, DRY scope, import aliases, public-method contract, extension points, evolution checklist, out-of-scope surfaces, and §10 ALLOWED / FORBIDDEN path tables. Required reading before any Plan or Code subagent that touchestests/components/,api/schemas/, or fixtures.../agentic-qa-core/references/skill-composition-strategy.md— T1/T2/T3/T4 tier model, category vocabulary, validation rules. The §4 anti-leak contract is informational here: framework-development no longer chains SDD by default; §4 governs users who manually install SDD and explicitly request the SDD ceremony.../agentic-qa-core/references/briefing-template.md— 7-component briefing examples per pattern.../agentic-qa-core/references/dispatch-patterns.md— Single / Sequential / Parallel / Background decision guide.../agentic-qa-core/references/orchestration-doctrine.md— failure protocol, ASK-on-error rule, no auto-fix.../agentic-qa-core/references/session-management.md— Phase 0 resume contract,plan.md/progress.mdschemas, archive policy, scope-naming, Engram coupling. This skill is one of the producers ofsession/...topic keys.
Files (agentic-qa-boilerplate)
-
references
-
kata-invariants.md 25.3 KB
# KATA Invariants — Framework Evolution Reference Canonical knowledge source for `framework-development`. Distinguishes what is INVARIANT (cannot break without major version bump) from what is EXTENSIBLE (safe evolution surface). Derived from the test-automation skill references and `docs/methodology/kata-fundamentals.md`. Use this file to gate framework changes: reject violations, approve safe extensions. Terminology preserved verbatim: ATC, fixture, locator, Component, Steps, Helper, Page, Api, TestContext, ApiBase, UiBase, TestFixture. > **Canonical formula — the one sentence every KATA surface must agree with:** > > **KATA organises automation in four layers with a single direction of dependency: TestContext, > Base, domain Components and Fixtures. Steps is an optional intermediate layer between Components > and Fixtures. Test files consume the Fixtures — they are not a layer.** > > Short form: **"four named layers, plus optional Steps"**. Never publish a bare number. Banned on > every surface: "three layers", "five layers", "6 layers", "Test files" as a layer, and > the acronym expanded with a C ("Component Action Test Architecture"). The expansion is > **Komponent Action Test Architecture**: the K lives only in the name; the layer is "domain > Components", spelled normally. DRY zones that are NOT layers: `tests/utils/`, `tests/data/`, `config/`. > Consumers: `tests/e2e/`, `tests/integration/`. Mirrored verbatim in > `test-automation/references/kata-architecture.md`; if the two ever differ, that is the drift. --- ## 1. The 4 layers (INVARIANT) A higher layer may use a lower layer; never the inverse. Steps (3.5) is an optional intermediate layer between Domain (3) and Fixtures (4). Adding a new layer is FORBIDDEN — it breaks the inheritance chain assumed by every component. | Layer | WHERE (file path) | WHAT (one-line responsibility) | WHY it cannot be collapsed | |-------|-------------------|--------------------------------|----------------------------| | 1 — TestContext | `tests/components/TestContext.ts` | Global config, faker, environment accessors, logger; agnostic to API/UI. | Shared baseline both ApiBase and UiBase extend. Collapsing into Base couples HTTP with browser. | | 2 — ApiBase | `tests/components/api/ApiBase.ts` | HTTP helpers (`apiGET`, `apiPOST`, `apiPUT`, `apiPATCH`, `apiDELETE`) returning typed tuples; auth token state; Allure attach. | Owns `APIRequestContext`. Without it, every Api component re-implements HTTP. | | 2 — UiBase | `tests/components/ui/UiBase.ts` | Playwright helpers (`interceptResponse`, `waitForApiResponse`, storage state, attachments). | Owns `Page`. Without it, every Page component re-implements Playwright wiring. | | 3 — Domain (`{Resource}Api`, `{Page}Page`) | `tests/components/api/*Api.ts`, `tests/components/ui/*Page.ts` | Business logic surface. ATCs (`@atc('TICKET-ID')`) live here exclusively. | Maps 1:1 to TMS tickets. Collapsing into Base loses traceability and groups unrelated resources. | | 3.5 — Steps (optional) | `tests/components/steps/*Steps.ts` | Reusable ATC chains used as preconditions across 3+ tests in 3+ files. NOT decorated with `@atc`. | Without Steps, callers either duplicate chains or violate Rule 5 (ATCs calling ATCs). | | 4 — Fixtures | `tests/components/{ApiFixture,UiFixture,TestFixture}.ts` | DI entry point. Instantiates and exposes every Domain component to test files. Lazy initialization (no browser unless requested). Steps modules are NOT fixture-registered — tests instantiate them directly (`new {Domain}Steps(options)`). | Tests cannot reach a component unless it is registered here. Removing Fixtures breaks Playwright's `test.extend` integration. | A test file is NOT a layer — it is a consumer of Layer 4. --- ## 2. Fixture selection (INVARIANT) Fixture choice determines whether a browser opens. Wrong fixture = slow API tests or missing context. Selection rule cannot change without major version bump because it is part of every test file's import contract. | Fixture | Use when | Browser? | Why it exists | |---------|----------|----------|---------------| | `{ api }` | API-only integration tests under `tests/integration/**`. | No (lazy). | Allows fast HTTP-only suites; avoids paying Playwright browser launch cost. | | `{ ui }` | UI-only E2E tests with no API setup. | Yes. | Pure Playwright workflow; no `APIRequestContext` overhead. | | `{ test }` | Hybrid: API setup + UI action + API verification. | Yes. | Shares the same `request` and `page` between Api and Ui components — required for token propagation across both surfaces. | There is NO `{ steps }` fixture. Steps modules (reusable precondition chains repeated across 3+ tests in 3+ files) live in `tests/components/steps/`, extend `TestContext`, and are instantiated directly in the test (`const steps = new {Domain}Steps({ request })`). They eliminate ATC-to-ATC calls (Rule 5 violation) without forcing tests to repeat setup — but they never enter the fixture DI surface. Hard rule: never request `{ ui }` for an API-only test. Never request `{ api }` from an `tests/e2e/**` UI-only file. Hybrid tests MUST use `{ test }`, not destructure `{ api, ui }` from separate registrations (they would not share context). --- ## 3. ATC identity (INVARIANT) An ATC = Acceptance Test Case = complete mini-flow mapped 1:1 to a TMS ticket via `@atc('TICKET-ID')`. The four ATC sub-rules are non-negotiable — a method violating any of them is not an ATC and must be reclassified or refactored. - **Atomic mini-flow**: precondition → action → verification → assertions → return. NEVER a single `page.click()` or single `apiGET`. A read-only GET is a Helper (no `@atc`, optional `@step`), not an ATC. - **NEVER calls another ATC**: ATCs are atomic. Reusable chains live in the Steps module (Layer 3.5). An ATC calling `this.someOtherAtc(...)` is a CRITICAL reject. - **Max 2 positional params; 3+ → object param**: `fn(a, b, c, d)` is FORBIDDEN. Use `fn(args: Args)`. Applies to ATCs and to every Layer 2/3/3.5 method. - **Locators inline; extract only if used 2+ times**: locators default inline inside the ATC. Extract to `private readonly someLocator = () => this.page.locator(...)` arrow function on the class only when used in 2+ ATCs of the same component. NEVER extract to a separate `locators/*.ts` file. Additional ATC invariants from Rule 2 (TC Identity = Precondition + Action) and Rule 3 (Equivalence Partitioning): same precondition + same action = ONE TC regardless of how many fields it asserts; same expected output with different data = ONE parameterized ATC. Naming pattern (INVARIANT): `{verb}{Resource}{Scenario}` camelCase. Suffixes: `Successfully` / `WithValidCredentials`, `WithInvalid{X}`, `WithNonExistent{X}`, `WithExpired{X}`, `WithRestricted{X}`. ATC class composition: max 15–20 ATCs per Domain component; split when larger. --- ## 4. DRY scope (INVARIANT) Where each kind of code MUST live. Misplacement creates coupling and fragile tests. The placement rule is structural — moving HTTP helpers into `tests/utils/` (because "they are utilities") breaks the architecture. | Code kind | MUST live in | MUST NOT live in | |-----------|--------------|------------------| | OpenAPI type facades (re-exports of `components`/`paths`) | `api/schemas/{domain}.types.ts` | Domain components (only facades may import `@openapi`) | | Agnostic utilities (no Playwright, no HTTP) — Allure attach helpers, string formatters, validators | `tests/utils/` | UiBase, ApiBase, Domain components | | ALL Playwright helpers (anything needing `Page` / `PageContext`) — `interceptResponse`, `waitForApiResponse`, storage-state snapshots | `UiBase` (`tests/components/ui/UiBase.ts`) | `tests/utils/`, TestContext, Domain components | | ALL HTTP helpers (anything needing `APIRequestContext`) — `apiGET/POST/PUT/PATCH/DELETE`, auth token state | `ApiBase` (`tests/components/api/ApiBase.ts`) | `tests/utils/`, TestContext, Domain components | | Cross-both shared (faker, config accessors, environment selection) | `TestContext` (`tests/components/TestContext.ts`) | UiBase, ApiBase (anything Playwright/HTTP-specific) | | Domain-specific logic for one resource/page | Layer 3 component (e.g. `generateOrderPayload()` inside `OrdersApi`) | Base classes, utilities | | Reusable ATC chains for preconditions (3+ ATCs across 3+ files) | `tests/components/steps/{Domain}Steps.ts` | ATC body (would violate Rule 5) | | Test data generators (faker-backed, typed) | `tests/data/DataFactory.ts` (+ types in `tests/data/types.ts`) | Direct `faker` import in components or tests | | Reference fixture data (roles, mock responses, parameterised inputs) | `tests/data/fixtures/*.json` (committed) | Hardcoded literals in tests | If you are tempted to put an API helper in `tests/utils/`, stop — it depends on `APIRequestContext`, so it belongs in `ApiBase`. --- ## 5. Import aliases (INVARIANT) Aliases are mandatory across `tests/**`, and **this one is now a compiler, not just doctrine**. The alias set is declared in `tsconfig.base.json`; `KATA_IMPORT_ALIASES` in `eslint.config.base.js` is a core `no-restricted-imports` block scoped to `tests/**/*.ts` + `playwright.config.ts` that rejects every `./` and `../` import there. It is a SECOND block beside `CLI_IMPORT_CLOSURE` (which stays scoped to `cli/**` and guards the updater's import closure); the two file sets are disjoint. Three things a reader should know before citing it: - There is still **no `eslint-plugin-import`** in this repo. The rule is core ESLint. Do not attribute it to a plugin nobody installed — that was the previous version of this paragraph's mistake, in reverse. - **Dynamic `await import('./x')` is not caught.** The rule matches static import and export declarations only. - **`eslint.config.js` is project-owned and never overwritten by the sync**, so a downstream project receives the exported block and not the wiring. `validateEslintBlockWiring` (`cli/lib/agent-compatibility-contracts.ts`) fails `agents:compat:check` when a block the base exports is absent from the consumer, which is what stops the rule from shipping inert. Review (`/pr-review-lead`) is no longer the only enforcement point, but it still owns the half a lint rule cannot judge: whether the alias chosen is the RIGHT one for the layer. The alias set actually declared in `tsconfig.base.json` `paths` (the authority — read it, do not trust a copy; `tsconfig.json` extends the base and declares no `paths` of its own): ``` "@/*" -> ./* "@ui/*" -> ./tests/components/ui/* "@api/*" -> ./tests/components/api/* "@steps/*" -> ./tests/components/steps/* "@utils/*" -> ./tests/utils/* "@data/*" -> ./tests/data/* "@variables" -> ./config/variables.ts "@TestContext" -> ./tests/components/TestContext.ts "@UiFixture" -> ./tests/components/UiFixture.ts "@ApiFixture" -> ./tests/components/ApiFixture.ts "@TestFixture" -> ./tests/components/TestFixture.ts "@DataFactory" -> ./tests/data/DataFactory.ts "@openapi" -> ./api/openapi-types.ts (FACADE-ONLY consumer) "@schemas/*" -> ./api/schemas/* "@schemas" -> ./api/schemas/index.ts ``` There is no `@config/*` and no `@components/*`: config is reached through `@variables`, and the component tree through the per-layer aliases (`@ui/*`, `@api/*`, `@steps/*`) or the named fixture / context entries. Earlier revisions of this file listed both; they never existed in `tsconfig.json`, and a framework change that assumes them will not resolve. Rule: Domain components import from `@schemas/{domain}.types`, NEVER from `@openapi`. Only files under `api/schemas/` may import `@openapi`. Test files import `test` from `@TestFixture`, NOT from `@playwright/test`. Renaming an alias is a major-version-bump-level change because every test file and component imports through it. --- ## 6. Public method contract (INVARIANT) KATA distinguishes the public API surface (ATCs and class-public helpers) from internal utilities. Error-handling discipline differs by surface and is part of the test contract. - **Methods a test can reach (ATCs, class-public `@step` helpers): fail fast — `throw new Error(...)` with descriptive message**. Test must fail loudly at the call site. Note the `apiXXX` HTTP primitives are `protected`, not part of that surface: `ApiBase` fails fast from its `get request()` getter when no context is available. - **Private utilities (parsers, matchers, internal helpers): silent fail — `return null` / `return undefined`**. Caller decides. Example: `parseResponseBody<T>` returns `null` when the response is not JSON. - **Why**: public methods are part of the test contract that downstream tests rely on; failing silently masks real bugs. Utilities are convenience — `null` lets the caller handle the missing-data case explicitly. Additional public-surface invariants: - Never swallow an error inside an ATC without re-throwing. - Sensitive parameter names must stay canonical (`password`, `token`, `secret`, `authorization`, `access_token`) so `@atc` / `@step` decorators auto-mask in trace output. Renaming these keys silently leaks credentials to Allure / NDJSON. - Tuple-return contract per HTTP method is INVARIANT: `apiGET/DELETE` → `[APIResponse, T]`; `apiPOST/PUT/PATCH` → `[APIResponse, T, P]`. Changing tuple shape breaks every Domain ATC. --- ## 7. Extension points (EXTENSIBLE — explicit allowlist) Where new code CAN safely land WITHOUT a major-version bump. Anything not on this list requires architectural review. | Extension | Where | Constraint | |-----------|-------|------------| | New agnostic utility | `tests/utils/<name>.ts` | MUST NOT depend on Playwright `Page` or `APIRequestContext`. If it does, it belongs in `UiBase` or `ApiBase`. | | New helper method on `ApiBase` | `tests/components/api/ApiBase.ts` | MUST be reusable across multiple `*Api` subclasses. Domain-specific logic stays in the Domain component. | | New helper method on `UiBase` | `tests/components/ui/UiBase.ts` | Same constraint — must be reusable across multiple `*Page` subclasses. | | New Domain component | `tests/components/api/{Resource}Api.ts` or `tests/components/ui/{Page}Page.ts` | MUST be registered in the matching Fixture (`ApiFixture`, `UiFixture`). Without registration, tests cannot reach it. ApiFixture must also forward `setAuthToken` and `clearAuthToken` to it (there is no `setRequestContext` — the request context arrives through the constructor). | | New Steps module | `tests/components/steps/{Domain}Steps.ts` | Extends `TestContext`; instantiated directly in tests (`new {Domain}Steps(options)`) — NEVER fixture-registered. NOT decorated with `@atc`. Used only when 3+ ATCs repeat across 3+ files. Anti-duplication check: `kata-manifest.json` `steps[]` lists every existing Steps module — consult it before proposing a new one. | | New Fixture registration entry | `ApiFixture` / `UiFixture` constructor | Must mirror the auth propagation pattern (forward `setAuthToken` and `clearAuthToken` from the ApiFixture overrides; the request context is constructor-injected, not forwarded). | | New OpenAPI facade | `api/schemas/{domain}.types.ts` | Must be re-exported from `api/schemas/index.ts` barrel. Only facade files import `@openapi`. | | New DataFactory generator | `tests/data/DataFactory.ts` (+ matching interface in `tests/data/types.ts`) | Use `faker` only inside DataFactory. Tests/components must NEVER import `faker` directly. | | New static fixture data | `tests/data/fixtures/*.json` | Only for reference data (roles, permission matrices, mock responses, configuration trees). Transactional data goes to DataFactory. | | New script | `scripts/<name>.ts` | Add the matching `bun run` entry to `package.json`. | | New CLI command | `cli/<command>/` | Project-level installer concern; standalone binaries do not affect the runtime test architecture. | | New TS path alias | `tsconfig.json` `paths` | Allowed for new layers/folders — but never collapses an existing alias. | | New Playwright tag | usage in `test()`/`describe()` | Must be documented in `automation-standards.md` §4 tag table. | --- ## 8. Evolution patterns (CHECKLIST — "if you change X, verify Y") Mandatory verification matrix when modifying load-bearing surface area. Each row gates a CRITICAL change. | If you change... | You must verify... | |------------------|-------------------| | `ApiBase.apiGET/POST/PUT/PATCH/DELETE` signature or tuple return | Re-run ALL Api ATC tests; type-check entire repo; grep every `*Api.ts` for tuple destructure shape. | | `ApiBase` auth methods (`setAuthToken`, `clearAuthToken`) | Verify `ApiFixture` overrides forward to every registered Api component. Run all 401-coverage tests. | | `UiBase.interceptResponse` / `waitForApiResponse` signature | Re-run ALL UI ATCs that use interception; confirm Allure attachments still produce. | | `TestContext` constructor or option shape (`TestContextOptions`) | Audit every Layer 2/3/3.5 constructor that calls `super(options)`. Re-run full suite. | | Fixture signature in `TestFixture`/`ApiFixture`/`UiFixture` | Grep all consumers (`tests/**/*.test.ts`); update destructures; re-run full suite. | | Import alias in `tsconfig.json` paths | Update tsconfig + every import in repo + ESLint config. Run `bun run types:check` + `bun run lint:check`. | | `@atc` / `@step` decorator API or `SENSITIVE_KEYS` set | Re-run full suite; manually inspect Allure step titles for unmasked sensitive values; verify NDJSON line schema unchanged. | | `KataReporter` NDJSON line schema or `atc_results.json` aggregation logic | Verify teardown summary still parses; verify TMS sync (`syncToXray`, `syncToJiraDirect`) still consumes correct fields. | | `tests/utils/decorators.ts` `storeResult` writer | Confirm NDJSON file is still atomic-append safe; confirm reporter `onEnd` deletes the partial file. | | `kata:manifest` extraction regex (`@atc\s*\(\s*['"]([^'"]+)['"]`) | Verify it still matches every existing `@atc('...')` call. Computed/template-literal IDs remain unsupported by design. | | OpenAPI facade pattern (replace `paths`/`components` indexing) | Regenerate types via `bun run api:sync`; re-run `bun run types:check`. | | Adding a new layer between existing layers | FORBIDDEN — breaks the 4-layer model invariant. Reject. | | Replacing the Playwright dependency | Major-version bump; UiBase entire surface invalidated. Reject without version-bump approval. | --- ## 9. What `framework-development` CANNOT touch Out-of-scope surfaces. Modifying these from a framework-development task is FORBIDDEN — they belong to other workflows or are auto-regenerated. - **Generated artifacts**: `api/openapi-types.ts` is generated by `bun run api:sync` from `api/openapi.json`. Never hand-edit. `api/openapi.json` and `api/.openapi-config.json` are gitignored local cache. `kata-manifest.json` is generated by `bun run kata:manifest`. `reports/atc_results.json` and `reports/.atc_partial.ndjson` are runtime artifacts. - **Per-ticket test specs** (consumers, not framework): `tests/e2e/**/*.test.ts`, `tests/integration/**/*.test.ts`, and ticket-specific subclasses under `tests/components/{module}/` (the BASE classes `ApiBase`/`UiBase`/`TestContext` are framework; concrete `*Api`/`*Page` for a ticket are consumer code owned by `test-automation`). - **Per-ticket QA context**: `.context/PBI/**` (PBI folders, ATPs, ATRs, evidence). Owned by `sprint-testing` / `test-documentation`. - **Project-wide context**: `.context/business/**`, `.context/master-test-plan.md`. Owned by `/business-*-map`, `/master-test-plan`. (TMS modality + Regression Epic are resolved live by `/test-documentation` from `.agents/project.yaml` and Jira itself — no `.context/` file.) - **Credentials and env**: `.env`, `.env.example` (only the variable list may be appended when adding a new framework env var; never values). - **Playwright artifacts (gitignored)**: `test-results/`, `tests/data/downloads/`, `playwright/.auth/`. - **Test results / TMS sync state**: outputs of CI runs, not framework code. - **Skills / Commands / AGENTS.md**: owned by `/agentic-qa-core`, `/sync-ai-memory`, and the SDD orchestrator. A framework change that needs to surface in AI memory must coordinate via `/sync-ai-memory`, not direct edit. --- ## 10. ALLOWED and FORBIDDEN paths (POLICY) Two complementary tables that gate every `framework-development` change. The ALLOWED table enumerates the surface this skill owns; the FORBIDDEN table redirects to the owning workflow skill. Phase 0 of `framework-development/SKILL.md` cites this section by name — keep tables flat and grep-friendly. These are POLICY tables, not INVARIANT rules. They can be amended additively without a major-version bump (e.g. when a new layer or fixture type is introduced), but only via a `/framework-development` change that touches this file itself. ### 10.1 ALLOWED paths (framework surface) | Path | Why it lives here | |-------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| | `cli/` | Installer + agents:setup + vars:check — project-level tooling, ships with every clone | | `scripts/` | `bun run` script implementations (`api:sync`, `kata:manifest`, `jira:sync-fields`, `lint:skills`, etc.) | | `.agents/` (structure changes only) | Schema for `project.yaml`, `jira-fields.json`, `jira-workflows.json`, `jira-required.yaml`. Values stay manual. | | `tests/utils/` | Agnostic utilities — Allure attach helpers, decorators, formatters. Evolution of the utility layer. | | `tests/components/` (Layer 2 + 3 base classes only) | `TestContext.ts`, `ApiBase.ts`, `UiBase.ts`. NOT per-module `*Api.ts` / `*Page.ts` (those are test-automation). | | `tests/components/` (fixture files) | Fixture registry evolution — `ApiFixture.ts`, `UiFixture.ts`, `TestFixture.ts`. New fixture APIs. | | `scripts/sync-openapi.ts` and the sync pipeline | Pipeline source. NOT generated `api/openapi-types.ts` (that is regenerated by `bun run api:sync`). | | `package.json` deps + scripts | Dependency upgrades, script registry, engines. Not test specs in `tests/`. | | `.agents/skills/agentic-qa-core/references/` | Briefing template, dispatch patterns, orchestration doctrine, skill-composition-strategy. | | `.agents/skills/framework-development/` | This skill itself — references, scripts, agents/. | | `.claude/commands/` | Slash-command source (`/sync-ai-memory`, `/business-*-map`, `/master-test-plan`, etc.). | ### 10.2 FORBIDDEN paths (redirect map) | Path | Owned by | |-----------------------------------------------------|------------------------------------------------------------------------------------------------| | `tests/e2e/` | `/test-automation` — per-ticket E2E specs | | `tests/integration/` | `/test-automation` — per-ticket API/integration specs | | `tests/components/{module}/` (Page/Api/Steps) | `/test-automation` — module-specific Domain components and Steps | | `.context/PBI/` | `/sprint-testing` — per-ticket QA context | | `.context/master-test-plan.md` | `/master-test-plan` command — regenerative | | `.context/business/` | `/business-data-map`, `/business-feature-map`, `/business-api-map` commands — regenerative | | `api/openapi-types.ts` | Generated artifact — regenerated by `bun run api:sync` | | `kata-manifest.json`, `reports/atc_results.json` | Generated artifacts — runtime / build outputs | | `.env`, credentials | Manual edit only — no skill, no AI rewrite | ### 10.3 Tie-break rules - A path matches neither table → ASK the user explicitly. Never assume. - A single change spans both tables → SPLIT-WORK: framework-development owns the framework slice; the FORBIDDEN-row owner owns the consumer slice in a follow-up. - Hot-spot exceptions: changes to `tests/components/{module}/` that ONLY rename or move existing files as part of a base-class evolution are still owned by `/test-automation` — coordinate via the SPLIT-WORK rule rather than reaching for the file from this skill. --- *Reference compiled from `kata-architecture.md`, `typescript-patterns.md`, `api-patterns.md`, `e2e-patterns.md`, `atc-tracing.md`, `automation-standards.md`, `data-testid-strategy.md`, `test-data-management.md`, `review-checklists.md`, `test-automation/SKILL.md`, and `docs/methodology/kata-fundamentals.md`. Update only when a source doc changes a load-bearing rule.*
-
-
SKILL.md 28.6 KB
--- name: framework-development description: "Framework evolution mode — evolves the QA boilerplate itself (KATA, fixtures, cli/, scripts/, api/schemas/ pipeline, package.json deps). Self-contained Plan → Code → Verify → Archive pipeline; runs under the `gentle-ai install --preset minimal` install (no SDD-* skills required). Use when adding new fixture APIs, refactoring KATA base classes, evolving the installer, modifying the OpenAPI sync pipeline, or any change to the framework infrastructure that is NOT per-ticket test writing or manual QA. Triggers on: /framework-development, \"evolve framework\", \"framework refactor\", \"new fixture API\", \"modify KATA base\", \"refactor cli\", \"boilerplate evolution\". Do NOT use for: writing tests for a ticket (use /test-automation), manual QA per ticket (use /sprint-testing), documenting test cases (use /test-documentation), running regression suites (use /regression-testing)." license: MIT compatibility: [claude-code, copilot, cursor, codex, opencode] complementary_categories: [framework-evolution, meta-skill] --- # Framework Development — Evolve the QA Boilerplate Gateway skill for changes to the framework itself: KATA layers, fixtures, installer, OpenAPI pipeline, scripts, doctrine docs. Per-ticket QA work, test specs, and TMS documentation are owned by other workflow skills (`/sprint-testing`, `/test-documentation`, `/test-automation`, `/regression-testing`) and MUST NOT trigger this skill. The skill exists because framework-surface changes — new fixture, new layer helper, installer rewrite, manifest extractor — deserve a planning gate before code. Per-ticket test writing already has its own gate in `/test-automation` Plan → Code → Review; this skill is its architectural-surface counterpart. --- ## Compact Rules - DO NOT: use this skill for per-ticket work — test writing is `/test-automation`, manual QA is `/sprint-testing`, TMS docs are `/test-documentation`, suite runs are `/regression-testing`. This skill governs the architectural surface only. - DO: clear the readiness preflight, then run the Phase 0 path self-check against `references/kata-invariants.md` §10 before dispatching anything. A FORBIDDEN path aborts and redirects to the skill named in the row; a path in neither table is ASKED about, never assumed. - WHEN one change spans both ALLOWED and FORBIDDEN paths: split it. This skill changes the base; `/test-automation` migrates the consuming specs in a follow-up. - DO: run Plan → Code → Verify → Archive in order for every non-trivial framework change. The pipeline IS the gate; "it's a quick refactor" is not an exemption. - DO NOT: edit `tests/components/` from a framework-development session — those L2/L3 KATA components are per-ticket surface. - DO NOT: collapse the KATA layers (TestContext / Base / Domain / Fixture) under a simplicity argument. They are framework architecture, not speculative abstraction. - DO NOT: add a new fixture API without updating the matching fixture file AND `kata-manifest.json` AND citing at least one existing test that consumes it. Orphan fixtures rot, and the manifest is the anti-duplication gate. - DO NOT: bump a major version of Playwright / Bun / TypeScript without a regression run on a representative E2E suite — lockstep upgrades hide breaks in fixture lifecycle, locator engines, and type emit. - DO NOT: refactor `cli/install.ts` without exercising the full install flow on a clean clone. Verification on an already-installed repo proves nothing, and the installer is the one surface where a bug ships silently to every new user. - WHEN the chosen approach reshapes test architecture (KATA layers, a fixture API, the runner, the isolation/parallelization model, the OpenAPI/type pipeline) AND is hard to reverse: record an ADR under `.context/ADR/` after plan approval and before coding, drafted `Proposed` for the human to accept. ADRs are append-only — supersede, never rewrite. - DO: verify with all four checks (test, types, lint, skills) and treat any non-zero exit as REJECT — present retry / skip-and-document / abort, never auto-fix. - DO NOT: let a subagent write `progress.md`; it is orchestrator-only. Code subagents return one-line summaries per task, and the orchestrator does not read their diffs. - DO: archive the session directory only after all four verifiers pass. On REJECT it stays in place so the run can be debugged or resumed. **Read full SKILL.md when**: writing the plan artifact, batching Code-phase tasks, resuming an interrupted session, or reading the ALLOWED/FORBIDDEN path tables themselves. --- ## Inputs Canonical reading order for any AI starting cold on a framework-development workflow. Read in order; stop earlier when the change is small enough that later inputs add no signal. 1. `kata-manifest.json` — Component + ATC registry (source of truth per Critical Rule #12). Establishes what already exists before any new fixture API, Page, Api, Steps module, or ATC ID is proposed. 2. `.agents/skills/test-automation/references/kata-architecture.md` + `.agents/skills/test-automation/references/typescript-patterns.md` — KATA layer flow (TestContext → ApiBase / UiBase → YourApi / YourPage → TestFixture), ATC identity rules, fixture-selection contract, import-alias conventions. 3. `tests/components/` — current Api / Page / Steps shape; required reading when touching any L2 / L3 surface or adding a fixture consumed by these components. 4. `cli/install.ts` — installer flow; required reading when evolving the installer, adding install steps, or modifying boilerplate scaffold behavior. 5. `scripts/sync-openapi.ts` + `api/schemas/` — OpenAPI-derived TypeScript types pipeline; required reading when touching the API contract pipeline, schema generation, or any consumer of generated facades. 6. `package.json` + `bun.lockb` — dep landscape; required reading before bumping Playwright / Bun / TypeScript / fixture-runtime versions or adding/removing scripts. --- ## Subagent Dispatch Strategy > **Orchestration & Session contracts**: this skill follows `agentic-qa-core/references/orchestration-doctrine.md` (mandatory subagent dispatch — main thread is command center) AND `agentic-qa-core/references/session-management.md` (Phase 0 resume check, plan-first persistence at `.session/<skill-slug>/<scope>/`, archive on completion). Phase 0 (resume check) and Phase 1 (plan write) are NOT optional. This skill is compliant with the doctrine in `AGENTS.md` §"Orchestration Mode (Subagent Strategy)" and the session contract in `.agents/skills/agentic-qa-core/references/session-management.md`. Every dispatch follows the 7-component briefing format defined in `.agents/skills/agentic-qa-core/references/briefing-template.md`, and the pattern selected per phase matches the decision guide in `.agents/skills/agentic-qa-core/references/dispatch-patterns.md`. The four phases — Plan, Code, Verify, Archive — mirror the shape of `/test-automation` (Plan → Code → Review) extended with an inline Archive step. Phase 0 stays inline because the path self-check + session resume check are short orchestrator decisions that do not benefit from a fresh-context subagent. **Session scope**: `<change-name>` (kebab-case, user-provided at session start). Session state lives at `.session/framework-development/<change-name>/{plan.md, progress.md}` per `agentic-qa-core/references/session-management.md` §9. | Phase | Pattern | Subagent role | |----------------------------------------------------|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| | Phase 0 — Path self-check + session resume check | inline | orchestrator only; no subagent. Lists target paths against `references/kata-invariants.md` §10; aborts on FORBIDDEN. Also checks `.session/framework-development/<change-name>/` for prior plan/progress and offers resume per `agentic-qa-core/references/session-management.md` §4 | | Phase 1 — Plan (`plan.md`) | Single | one Plan subagent writes `.session/framework-development/<change-name>/plan.md` per `agentic-qa-core/references/session-management.md` §6 schema; collapses prior explore + propose + spec + design + tasks into one artifact | | Phase 2 — Code (per task batch) | Sequential | one Code subagent per task batch from the plan; orchestrator appends to `progress.md` per `agentic-qa-core/references/session-management.md` §7 schema; on verification failure: STOP, no auto-fix | | Phase 3 — Verify — `bun run test` | Parallel (sub-stage) | one Verifier subagent runs the test suite | | Phase 3 — Verify — `bun run types:check` | Parallel (sub-stage) | one Verifier subagent runs typecheck | | Phase 3 — Verify — `bun run lint:check` | Parallel (sub-stage) | one Verifier subagent runs ESLint | | Phase 3 — Verify — `bun run skills:check` | Parallel (sub-stage) | one Verifier subagent runs the skill-registry lint (framework changes can affect `.agents/skills/`, `AGENTS.md`, `cli/install.ts`) | | Phase 3 — Aggregation + accept/reject decision | inline | orchestrator reads the 4 Verifier reports and decides; on any non-zero exit, presents retry / skip / abort | | Phase 4 — Archive (move plan + progress) | inline | orchestrator only; moves `.session/framework-development/<change-name>/` to `.session/.archive/<YYYY-MM-DD>-framework-development-<change-name>/` (two-file dir preserved per `agentic-qa-core/references/session-management.md` §8); references in commit | - **Plan artifact location**: `.session/framework-development/<change-name>/plan.md`. The `.session/` tree is gitignored — the plan is local, not committed. Recovery on mid-run crash: the file persists; the orchestrator reads it back on the next session via Phase 0 resume check (see `agentic-qa-core/references/session-management.md` §4). - **Grace period for legacy path**: prior versions wrote to `.scratch/framework-changes/<change-name>/{plan.md, apply-progress.md}`. Phase 0 also checks the legacy path during the grace period — if found, the orchestrator offers to copy state to the new `.session/...` location before resuming. - **Path guardrails injected per dispatch**: every Plan and Code subagent briefing MUST include the line `KATA invariants and ALLOWED/FORBIDDEN paths: .agents/skills/framework-development/references/kata-invariants.md (read §10 before touching any file).` Do NOT inline the path tables — the reference is authoritative. - **On any subagent failure**: STOP, return the failing report, do NOT auto-rerun. The orchestrator decides retry / skip / abort. See `.agents/skills/agentic-qa-core/references/orchestration-doctrine.md`. - **Strict TDD flag** is set in Phase 1's `plan.md` under §"Strict TDD flag". Default OFF. Flipped ON only when the user explicitly opted in. Code phase reads it from the plan; no separate cache needed. --- ## Fleet seam (optional) Phase 2 Code is sequential per task batch by default, and a change that fits one plan stays that way. A **wave** — several persistent worker sessions taking one task batch each, a conductor integrating them — is for a plan whose batches are genuinely independent (different files, no shared public API in flight). Offer it; never enter it silently. - **Topology: same checkout with file ownership per worker.** Each worker's brief lists the exact files it owns and commits with explicit paths (never `git add -A`), so two workers never stage each other's work. When two batches must touch the same module, they do not run in the same wave: serialize them, or give each worker its own worktree. - The conductor writes `launch.txt` in `.session/framework-development/<change-name>/` — one self-contained line per worker — **always**, whether or not any orchestration transport exists on the machine. Launching, supervising and closing those sessions is `orca-orchestration/SKILL.md` (`[ORCHESTRATION_TOOL]`): supervised launch is the native path, and `launch.txt` is the payload for the human-paste fallback when nothing can launch it. - **Phase 0 and Phase 3 stay with the conductor.** The path self-check runs once, over the union of every worker's paths, before the first launch; the four Phase 3 verifiers run once, on the integrated tree, after the last worker reports. A per-worker green gate is not a wave gate. - **Read before every edit.** A worker's neighbour may have changed a shared file since the brief was written; Critical Rule #15 (no global discards) binds twice as hard when sessions share a tree. - **Silence rule**: the absence of an orchestration transport is never named to the user and never appears in the preflight gate or the plan. --- ## Readiness Preflight Gate (MANDATORY — runs before Phase 0) > Full doctrine: `agentic-qa-core/references/preflight-gate.md`. Runs FIRST, before the path self-check and resume check. Two laws: (1) **args-as-answers** — the change name and touched paths are provided args; ask only the gaps. (2) **probe, don't assume**. Surface gaps + REDs as ONE `AskUserQuestion` checklist; self-fix with approval + explanation; STOP on any blocking RED. This skill evolves the framework itself — it does NOT hit a live env, Jira, DB, or API — so its gate is a **dev-toolchain readiness** check that pairs with the Phase 0 path self-check. **Generic baseline** (the two laws, secret/restart handling, output contract) is inherited from the reference §3.1 — not repeated here; the env/creds half of the baseline is N/A for meta-work. Below is only this skill's **specific capability delta**. | Capability | Need | Why here | |---|---|---| | Dev toolchain | REQUIRED | Phase 3 Verify runs `bun run test` / `bun run types:check` / `bun run lint:check` / `bun run skills:check`. All four must resolve at t=0. `bun install` if a dep is missing. | | `kata-manifest.json` clean | REQUIRED | Source of truth (Critical Rule #12). Framework changes can invalidate it — `bun run kata:manifest:check` clean, `bun run kata:manifest` to regenerate. | | Playwright browsers | SCOPE — touching fixtures / KATA bases / tests | Verify of a fixture or base-class change runs the suite, which needs chromium (`bun run pw:install`). | | `/github-actions-docs` + `/playwright-best-practices` | OPTIONAL | Injected per dispatch when the change touches CI YAML or fixtures/tests (already noted in the briefing skeleton). | Active env, test-user creds, OpenAPI/`API_TOKEN`, DBHub, issue-tracker, TMS and `resend` are **N/A** — framework evolution is meta-work on this repo. After the gate clears (all REQUIRED GREEN), continue to Phase 0 below. --- ## Phase 0 — Path self-check + session resume check (mandatory, runs first) Before invoking any subagent, the orchestrator MUST (a) list the files / directories the change will touch and verify each one against the ALLOWED / FORBIDDEN tables in `references/kata-invariants.md` §10, and (b) run the session resume check per `agentic-qa-core/references/session-management.md` §4. Skipping Phase 0 is the most common way framework changes leak into ticket-owned surface area OR lose mid-run state on interruption. 1. Read `references/kata-invariants.md` §10 (ALLOWED + FORBIDDEN paths). 2. Ask the user (or infer from the request): "Which paths will this change touch?" 3. For each path, look it up in §10 ALLOWED → proceed. Or §10 FORBIDDEN → abort and redirect to the skill named in the row. 4. If a path matches neither table, ASK the user explicitly — never assume. 5. If a single change spans both ALLOWED and FORBIDDEN paths (e.g. "refactor `tests/components/ui/UiBase.ts` AND update the e2e tests that consume it"), split the work: framework-development handles the base-class change; `/test-automation` handles the test-spec migration in a follow-up. 6. **Session resume check** (per `agentic-qa-core/references/session-management.md` §4): check `.session/framework-development/<change-name>/progress.md`. If it exists, read `plan.md` + the tail of `progress.md`, surface the last completed phase + next planned phase + any blocking notes, and offer **resume / restart / abort**. On `restart`, archive the current directory to `.session/.archive/<YYYY-MM-DD>-framework-development-<change-name>-aborted/` before proceeding. 7. **Legacy path check** (grace period): also check `.scratch/framework-changes/<change-name>/` for prior plan/progress under the old layout. If found, offer to migrate the state to the new `.session/...` location. Phase 0 is one short inline decision — it does NOT write a file. If the change is approved, the decision is captured later in Phase 1's `plan.md` §Investigation. --- ## Native Phase Orchestration After Phase 0 passes, run the four-phase pipeline in dependency order. Each subagent dispatch follows the 7-component briefing format in `agentic-qa-core/references/briefing-template.md`. Briefing skeleton for Plan and Code phases (fill the `<...>` slots): ``` Goal: <one-sentence outcome scoped to this phase> Context docs: - .agents/skills/framework-development/references/kata-invariants.md - .session/framework-development/<change-name>/plan.md (Code phase only) - .session/framework-development/<change-name>/progress.md (Code phase, batches > 1; orchestrator-written, read-only for subagents) - <relevant ALLOWED-path files the phase will read or touch> Project Standards (auto-resolved): <compact-rule blocks pulled from .agents/skills/REGISTRY.md per skill-resolver protocol> Skills to load: <none by default; orchestrator injects /playwright-best-practices if fixtures/tests, /github-actions-docs if CI YAML> Exact instructions: 1. Read kata-invariants.md fully. Verify §10 ALLOWED for every touched path. FORBIDDEN → STOP. 2. <phase-specific step — e.g. "write the plan", "implement task batch N"> 3. Save the artifact at the engram topic_key framework/<change-name>/<phase>. 4. Return the executive summary inline. Report format: - status: ready | blocked | failed - artifact: <absolute path or engram topic_key> - next_recommended: <phase or "stop"> - risks: [<one-liner per risk>] - skill_resolution: injected | fallback-inline Rules: - ALLOWED paths only (kata-invariants.md §10). FORBIDDEN → abort. - Do NOT modify generated artifacts (api/openapi-types.ts, kata-manifest.json, reports/). - If strict TDD is ON (read from plan §"Strict TDD flag"), every production-code task is preceded by a failing test in the same batch. - On uncertainty, STOP and report — do not improvise on framework surface. ``` Phase order (each phase gates the next): ``` Phase 0 (inline) -> Phase 1 Plan (Single) -> Phase 2 Code (Sequential per batch) -> Phase 3 Verify (Parallel 4-way) -> Phase 4 Archive (inline) ``` ### Phase 1 — Plan Dispatch: **Single**. The Plan subagent writes one consolidated artifact at `.session/framework-development/<change-name>/plan.md` covering: Goal, Investigation, Approach options, Chosen approach, Invariants touched, Public API delta, Task breakdown, Strict TDD flag, Risks, Verification checklist. One file, ten sections — not five separate documents. The seven base sections from `agentic-qa-core/references/session-management.md` §6 are mandatory; framework-development extends them with three skill-specific sections (Invariants touched, Public API delta, Strict TDD flag). Present the plan to the user. Wait for approval before Phase 2. **ADR seeding (framework architecture).** When the chosen approach reshapes the framework's test architecture — KATA layers, fixture APIs, the test runner, the isolation/parallelization model, or the OpenAPI/type pipeline — and the decision passes the two-gate test (architectural AND hard to reverse per `agentic-qa-core/references/adr-doctrine.md` §1), record a `.context/ADR/ADR-NNNN-<slug>.md` after the plan is approved and before Phase 2 coding. Framework evolution is meta-work: its decisions bind every test session that follows, so historicize them rather than leaving them in a one-off `plan.md` that gets archived. The plan's "Invariants touched" / "Public API delta" sections are the prime ADR candidates. Draft `Proposed`; the human accepts. Template + lifecycle: `.context/ADR/README.md`. ### Phase 2 — Code Dispatch: **Sequential** — one subagent per task batch. The orchestrator decides batching from the plan's task list; rule of thumb is 1 batch per 3-5 closely-coupled tasks, or 1 batch per task when the task touches a load-bearing file such as `ApiBase.ts` / `UiBase.ts` / `TestContext.ts`. Each subagent: 1. Reads `plan.md` and applies the tasks in its batch in plan order. 2. Runs the per-task verification command listed in the plan. 3. Returns a one-line summary per task to the orchestrator. 4. On verification failure: STOP, report, do not auto-fix. The orchestrator never reads diffs from the Code subagent — only the summary. If the user wants to see actual changes, the orchestrator runs `git diff` inline after the batch returns. After each batch returns, the orchestrator appends a phase entry to `.session/framework-development/<change-name>/progress.md` per `agentic-qa-core/references/session-management.md` §7 (subagents never write to `progress.md` directly — that is an orchestrator-only file). ### Phase 3 — Verify Dispatch: **Parallel** — four Verifier subagents in the same `<function_calls>` block: | Verifier | Command | Captures | |----------|------------------------|----------------------| | V1 | `bun run test` | exit code, summary | | V2 | `bun run types:check` | exit code, summary | | V3 | `bun run lint:check` | exit code, summary | | V4 | `bun run skills:check` | exit code, ERROR/WARN/INFO counts | `skills:check` is included because framework changes routinely touch `.agents/skills/framework-development/`, `agentic-qa-core/references/`, `AGENTS.md`, and `cli/install.ts` — every one of those surfaces is read by `scripts/lint-skills.ts` and gated by 10 named checks (tier coherence, anti-leak, stale-path, duplicate-tier, etc.). The other three commands never see this surface; adding the fourth verifier costs one parallel slot and prevents an entire failure class. After all four return, the orchestrator inline-aggregates: - All four `exitCode == 0` → ACCEPT. Proceed to Phase 4. - Any `exitCode != 0` → REJECT. Present failing verifier(s) to the user. Options: retry the failing Phase 2 batch / skip-and-document / abort. Do NOT auto-fix. ### Phase 4 — Archive Dispatch: **inline** — no subagent. The orchestrator performs the archive flow from `agentic-qa-core/references/session-management.md` §8: 1. Verifies the Verification checklist in `plan.md` passes (all four Phase 3 verifiers returned exit 0). 2. Moves the entire working directory: `mv .session/framework-development/<change-name>/ .session/.archive/<YYYY-MM-DD>-framework-development-<change-name>/`. Both `plan.md` and `progress.md` are preserved side by side (no concatenation) so future resume-replay stays possible. 3. Calls Engram `mem_session_summary` with the session template per `agentic-qa-core/references/session-management.md` §11. The summary MUST include the archive path so `mem_search "session framework-development <change-name>"` resolves back to the artifacts. 4. Surfaces the archive path so `/git-flow-master` can include it in the commit message body. Archive is a "close-the-loop" step, not "ship-the-code". Code is shipped by `/git-flow-master` based on the diff that Phase 2 produced and Phase 3 verified. On Phase 3 REJECT, archive does NOT run — the working directory stays in place so the user can debug, resume, or abort. --- ## Anti-patterns — NEVER do these - **F1.** NEVER use `/framework-development` for per-ticket test writing — that surface is owned by `/test-automation` (Plan → Code → Review on KATA + Playwright + TypeScript). Framework-development governs the architectural surface only. - **F2.** NEVER collapse KATA layers (TestContext / Base / Domain / Fixture) under the pretext of simplification. The layers are framework architecture, not speculative abstraction. Critical Rule #12-adjacent: simplicity-first does NOT apply to KATA. - **F3.** NEVER edit `tests/components/` from a framework-development session — those are L2 / L3 KATA components owned by per-ticket work via `/test-automation`. If a base-class refactor forces a consumer migration, split the work: framework-development changes the base; `/test-automation` migrates the specs in a follow-up. - **F4.** NEVER skip the Plan → Code → Verify → Archive pipeline for non-trivial framework changes. The pipeline IS the gate — bypassing it for "quick" refactors of `ApiBase.ts`, `UiBase.ts`, `TestContext.ts`, fixtures, installer, or OpenAPI pipeline reliably produces undetected regressions. - **F5.** NEVER bump major versions of Playwright / Bun / TypeScript without a regression run on a representative E2E suite. Lockstep upgrades hide breaking changes in fixture lifecycle, locator engines, or type-emit behavior. - **F6.** NEVER add a new fixture API without updating `tests/components/TestFixture.ts` (or the matching `ApiFixture.ts` / `UiFixture.ts`) AND `kata-manifest.json` AND citing at least one existing test that consumes it. Orphan fixtures rot — and `kata-manifest.json` is the anti-duplication gate (Critical Rule #12). - **F7.** NEVER refactor `cli/install.ts` without testing the full install flow on a clean clone. The installer is the only surface where a bug ships silently to every new user — verification on the developer's already-installed repo proves nothing. - **F8.** NEVER introduce a hard-to-reverse test-framework architectural decision (KATA-layer reshape, new fixture API, test-runner swap, isolation/parallelization model) without recording it as an ADR in `.context/ADR/`. Framework evolution binds every later test session — a decision left only in an archived `plan.md` gets re-litigated or silently violated. Draft `Proposed` before Phase 2; the human approves. ADRs are append-only: supersede, never rewrite. See `agentic-qa-core/references/adr-doctrine.md`. --- ## Session close contract **Session-footer contract (mandatory at close).** The final phase is not done until the two chat-facing blocks from `../agentic-qa-core/references/session-footer-contract.md` are printed: (1) consolidated screenshot list — repo-relative paths, verified on disk, bug annotations first — plus in-flow surfacing of every capture's path the instant it lands; (2) Session Footer listing skills/MCPs/CLIs actually used + testing levels touched, with explicit "none" entries for expected-but-untouched levels. Framing for this skill: meta. Multi-subagent sessions: each stage report carries the five footer fields (`skills_loaded`, `mcps_used`, `clis_used`, `testing_levels_touched`, `screenshots_captured`); the orchestrator compiles the footer ONCE at close. Chat only — never in a Jira comment or ATR body. --- ## References - `references/kata-invariants.md` — INVARIANT vs EXTENSIBLE rules for the 4 KATA layers, fixture selection, ATC identity, DRY scope, import aliases, public-method contract, extension points, evolution checklist, out-of-scope surfaces, and §10 ALLOWED / FORBIDDEN path tables. Required reading before any Plan or Code subagent that touches `tests/components/`, `api/schemas/`, or fixtures. - `../agentic-qa-core/references/skill-composition-strategy.md` — T1/T2/T3/T4 tier model, category vocabulary, validation rules. The §4 anti-leak contract is informational here: framework-development no longer chains SDD by default; §4 governs users who manually install SDD and explicitly request the SDD ceremony. - `../agentic-qa-core/references/briefing-template.md` — 7-component briefing examples per pattern. - `../agentic-qa-core/references/dispatch-patterns.md` — Single / Sequential / Parallel / Background decision guide. - `../agentic-qa-core/references/orchestration-doctrine.md` — failure protocol, ASK-on-error rule, no auto-fix. - `../agentic-qa-core/references/session-management.md` — Phase 0 resume contract, `plan.md` / `progress.md` schemas, archive policy, scope-naming, Engram coupling. This skill is one of the producers of `session/...` topic keys.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.