ui-verification
Runs scoped browser probes for focus, hit targets, overflow, themes, request failures, and performance attribution, with evidence linked to UI rule IDs. Use when asked to "verify this in the browser", "reproduce this finding", or "check the fix". For source audits and severity us
Install
npx skills add https://github.com/mblode/agent-skills/tree/main/skills/ui-verification
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mblode-agent-skills@llmmart
git clone https://github.com/mblode/agent-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mblode/agent-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
UI Verification
Owns the browser session. Every other UI skill in this repo reasons about source and infers what the user will see; this one loads the page and measures it.
- IS: booting the app, driving it with a browser, and running the probes that decide a rule at runtime: computed boxes, injected failures, observed layout shift, a scripted Tab walk, an axe scan per theme. Output is findings keyed to rule ids with reproducible evidence, plus the clearing re-run after a fix.
- IS NOT: finding defects by reading source or deciding their tier and ship verdict (
ui-designAudit mode owns both); building or restyling UI (ui-designBuild); authoring a durable test suite (write Playwright tests); pixel-diff regression against a baseline (Chromatic, Percy); field performance (RUM or CrUX; Lighthouse is a lab tool).
The division of labour is the point. A static audit reports what the code will probably do; it cannot see a 40px control whose hit area a pseudo-element already expands to 44, or a retry button wired to nothing. This skill reproduces or kills each of those, so a finding arrives with a measurement instead of a confidence.
Contents
- When to run
- Probe catalogue
- Progress checklist
- 1. Establish the session
- 2. Select the probe set
- 3. Run the probes
- 4. Decide each finding
- 5. Clearing re-run
- 6. Report
- Honesty rules
- Gotchas
- Related skills
When to run
| Situation | What this skill does |
|---|---|
A ui-design audit emitted findings and the user wants them confirmed |
Run only the probes those rule ids map to, in references/rule-coverage.md |
| No audit ran; the user points at a route or a running app | Detect features, run the full battery on the resolved routes |
| A fix just landed for a previously reproduced finding | Run the clearing re-run only (step 5) |
| The user asks for captures across themes or widths | probes/theme-locale-matrix.md alone |
A design-system.md claims a scale and someone needs it checked |
Read the claimed values off computed styles on a real page; a theme value the build overrides never reaches the browser |
Nothing here assigns a tier or a ship verdict. Hand reproduced findings back with their evidence and let ui-design's references/ship-readiness.md tier them; two skills tiering the same finding is how the tiers drift apart.
Probe catalogue
Each file is one probe: what it measures, the driver calls, the false positives it must guard, and the shape of the evidence it returns.
| Probe | Measures | Primary for |
|---|---|---|
| probes/axe-scan.md | axe-core violations per route and theme, including computed contrast | contrast, accessible names, landmarks, document language |
| probes/target-size.md | Bounding box and effective hit area of every visible interactive element | interaction-target-size |
| probes/focus-walk.md | Scripted Tab traversal, focus-ring pixel delta, dialog trap and restoration | focus-*, interaction-focus-visible, interaction-keyboard-operable |
| probes/layout-shift.md | Attributed layout-shift entries with the data response held open |
states-layout-shift, perf-image-dimensions-and-priority |
| probes/viewport-stress.md | Horizontal overflow and clipped text at 320px and up, and under tripled strings | layout-long-content-safety, mobile-viewport-scaling |
| probes/failure-injection.md | What renders when the data request returns 500, [], malformed, or nothing |
states-no-error-state, states-no-empty-state, async-*, microcopy-* |
| probes/theme-locale-matrix.md | The same route captured across viewport, theme, pseudo-locale, and direction | dark-i18n-untested, dark-i18n-rtl-untested |
| probes/console-network.md | Console errors, page errors, failed requests, hydration mismatch warnings | hydration mismatch, silent runtime failures |
| probes/web-vitals.md | LCP with its attributed element, CLS, INP on a scripted interaction | perf attribution, never a budget verdict |
Progress checklist
Verification progress:
- [ ] Step 1: Establish the session (references/session-setup.md): driver, build mode, base URL, auth, resolved routes
- [ ] Step 2: Select the probe set from handed-over rule ids (references/rule-coverage.md) or from the routes
- [ ] Step 3: Run each probe; record evidence artifacts before interpreting any of them
- [ ] Step 4: Decide each finding reproduced / not-reproduced / unknown; repeat timing-sensitive or inconsistent results when needed
- [ ] Step 5: For each fix applied, re-run the identical probe and record clearedBy
- [ ] Step 6: Emit the verification block per finding (references/evidence-output.md), then render
- [ ] Step 7: List probes skipped and why. A probe that could not run is never a pass
1. Establish the session
Read references/session-setup.md. It resolves four things, and getting any of them wrong invalidates every probe downstream:
- Driver. Playwright when the repo can run it, the Chrome DevTools browser tools when it cannot. Four probes need request interception and network control, so they do not exist under a driver without them.
- Build mode. Perf and layout-shift probes run against a production build; dev-mode numbers measure the bundler. Failure injection and focus probes run fine in dev.
- Auth and seeded data. A route behind a login with no seeded session returns
unknown, never a pass. - Routes. Map the diff to URLs. A component with no reachable route falls back to the repo's Storybook, and to
unknownif there is none.
Stop here if the app will not boot. A verification run with no session produces no findings, which is a reportable outcome and not a clean bill of health.
2. Select the probe set
Handed a list of rule ids (the normal case, from a ui-design audit): open references/rule-coverage.md, take the probe each id maps to, and run only those. Ids with no probe stay source-only findings and pass through untouched, marked as such.
Given only routes: detect features the way ui-design's feature playbooks do (form, list, modal, dashboard, checkout), then run the battery that surface earns. probes/axe-scan.md, probes/console-network.md, and probes/viewport-stress.md run on every route regardless: they are cheap, and they are the three that find things nobody suspected.
Budget the matrix before running it. Routes multiplied by viewports multiplied by themes grows fast, and a run that takes twenty minutes gets skipped next time. Two viewports (360 and 1280) and two themes cover the ground; add widths only where a probe already found an edge.
3. Run the probes
Each probe file carries its own recipe. Three rules hold across all of them:
- Capture evidence before interpreting it. Write the screenshot, the JSON measurement, and the console log to disk first. A finding whose evidence was never written is unverifiable by the person reading the report, which puts it back where the static audit left it.
- One probe, one route, one viewport, one theme. Never fold two conditions into one run: when the result surprises you, you need to know which axis produced it.
- Repeat uncertain measurements. Retry timing-sensitive or inconsistent results under controlled conditions. A deterministic failure with a captured trigger needs no duplicate run. A result that flips remains inconclusive until its conditions are understood.
4. Decide each finding
Three outcomes, and the middle one is the one that earns this skill its keep.
| Outcome | Meaning | What it does to the handed-over finding |
|---|---|---|
reproduced |
The probe measured the defect | Stays a fail, now carrying observed from the measurement rather than from the source read |
not-reproduced |
The tested conditions did not exhibit the defect | Withdraw only if the probe exercised the alleged trigger; otherwise retain the candidate with the remaining coverage gap |
unknown |
The probe could not run or could not decide | Finding survives as unknown with the probe's reason. Never converts to a pass |
A not-reproduced result is a real deliverable, not a wasted run. It is what removes the false positives a large rule corpus asserts with file:line confidence, and it feeds the rejection section ui-design already requires.
Where a probe finds something no static rule predicted, emit it as a new finding against the rule id the probe is primary for. Where no rule covers it (an axe violation with no ui-design counterpart, a console error), emit it under axe:<violation-id> or runtime:<signature> and say plainly that it came from the browser and not the corpus.
5. Clearing re-run
A fix is not verified by reading the diff. Re-run the identical probe: same route, same viewport, same theme, same seed, same injected failure. Record it as clearedBy on the finding, keep the before-artifact, and never overwrite it with the after.
Three outcomes worth naming:
- The probe now passes: the finding is
appliedand cleared. - The probe still fails: the fix did not work. Report it as applied-unverified and say so; do not report the fix and stay silent about the re-run.
- The probe passes but a different probe on the same route now fails: the fix caused a regression, which is a new finding, not a footnote on the old one.
6. Report
references/evidence-output.md owns the shape: a verification block appended to each finding, a top-level session block, and artifact paths. It defines only the delta on ui-design's references/output-adapters.md schema, which stays the single owner of the finding object, the three counts, and the verdict.
Where this skill runs standalone, render the same terminal adapter with SHIP VERDICT omitted: the verdict is a property of a tiered audit, and printing one from probe results alone invents a tier assignment nobody made.
Honesty rules
- A skipped probe is not a passed probe. Report every probe that did not run, with the reason (no route, auth required, driver lacks interception, app would not build). A report that silently omits them reads as coverage it does not have.
- A screenshot is evidence, not a verdict. Probes decide on measurements. Captures exist so a human can check the measurement, and for the two questions no measurement settles: whether the dark theme looks right, and whether the pseudo-locale broke the layout or merely the prose.
- Do not widen into a redesign. This skill reports what it measured. A finding that needs a new type scale names the mode to run next, exactly as an audit does.
- The app is the subject, not the harness. A failure caused by the probe itself (a selector that never resolved, a route interception that swallowed the wrong request) is a harness bug. Fix the probe and re-run; never report it as a defect in the app.
- Numbers carry their units and their conditions.
44x44px at 360px width with touch emulation onis a measurement.too smallis the inference this skill exists to replace.
Gotchas
The ones that cut across probes. Each probe file carries its own false positives.
page.emulateMedia({ colorScheme: 'dark' })does nothing for an app that themes with aclass="dark"ordata-themeattribute, which is most Tailwind apps. The probe reports a clean dark pass while never having left light mode. Drive the app's own toggle, and assert the attribute landed before capturing.- Running perf or layout-shift probes against
next devmeasures on-demand compilation. The first navigation to a route can spend seconds in the bundler, which lands in LCP and dwarfs anything real. - Animations that never settle keep a screenshot probe waiting until timeout. Set
prefers-reduced-motion: reducefor captures, then run motion-sensitive checks in a separate pass with it off, because reduced motion is also a code path that can be broken.
Related skills
ui-design: reads source, produces the findings this skill reproduces, and owns tiering, the ship verdict, and the finding schema.typography-audit: type findings that need a rendered measure or leading value can be handed here for the measurement.ax-audit: agentic surfaces. Its runtime questions use the same session and probes.ui-animation: motion craft. This skill can capture the timing, but judging the curve is that skill's.
Maintenance only: evals/evals.json holds the behavioural scenarios and routing prompts for anyone changing this skill. It never loads during a verification run.
Files (agent-skills)
-
evals
-
evals.json 5.6 KB
{ "skill_name": "ui-verification", "evals": [ { "id": 1, "prompt": "The ui-design audit flagged these on /settings and /billing: interaction-target-size on the close button in the API-keys dialog, states-no-error-state on the invoices table, and focus-not-restored on the delete-key confirm. Verify them in the browser. The app is running on :3000; /billing needs a logged-in user and there is no seeded session.", "expected_output": "A session block naming the driver, build mode, and the two routes with their status; the target-size and focus-walk probes run on /settings with measured observed values; every /billing finding returned as unknown with reason auth-required; no tier or ship verdict assigned to anything; the probesSkipped array populated rather than inferred from absence.", "files": [], "assertions": [ "Maps each rule id to its probe through references/rule-coverage.md rather than improvising a check", "Returns unknown with reason auth-required for the states-no-error-state finding on /billing; does not report it as pass, and does not report failure-injection as having run there", "Every reproduced finding carries route, viewport, and theme, and an observed value in units (px, not 'too small')", "Does not assign a tier, bump a tier, or print SHIP VERDICT; the findings are handed back for ui-design to tier", "Lists probesSkipped explicitly with a reason for each probe that did not run" ] }, { "id": 2, "prompt": "The audit says the icon button in TableRowActions.tsx line 41 fails interaction-target-size because it is h-8 w-8. Check it on a real page.", "expected_output": "The target-size probe run at a 360px touch viewport reporting a 32x32 bounding box whose sampled hit-test grid resolves to the button because a ::before pseudo-element expands the hit area, so the finding is not-reproduced and withdrawn into consideredAndRejected with the measurement as the guard.", "files": [], "assertions": [ "Measures at a touch viewport with touch emulation on, and says so in the conditions", "Reports both the bounding box and the effective hit area, not the bounding box alone", "Emits the result as not-reproduced and moves the finding to consideredAndRejected naming target-size and the measured hit area as the guard; does not silently drop it and does not leave it as a fail", "Does not require duplicate runs of a deterministic measured result; retries inconsistent measurements", "Cites an evidence artifact path that was written to disk before the verdict" ] }, { "id": 3, "prompt": "I fixed the invoices skeleton so it reserves height now. Did the fix actually work? Same route as before, /invoices.", "expected_output": "The layout-shift probe re-run under the identical conditions of the original finding (route, viewport, theme, held data response, production build), the before artifact preserved, a clearedBy block recorded on the finding, and any new failure on another probe for the same route reported as a separate new finding rather than folded into the old one.", "files": [], "assertions": [ "Re-runs layout-shift with the same route, viewport, theme, injected delay, and build mode as the original finding; does not compare a dev-build re-run against a production-build original", "Does not overwrite the before capture; the clearedBy evidence path is distinct from the original evidence path", "If the re-run still reproduces, reports the finding as applied with clearedBy.result reproduced and says the fix did not clear it, rather than reporting the fix and omitting the re-run", "If a different probe now fails on /invoices, emits it as a new finding, not a note on the cleared one", "Does not re-run the whole battery; the clearing re-run is scoped to the probe that produced the finding" ] }, { "id": 4, "prompt": "A static race finding occurs when a second request overtakes the first. A browser probe tested only one request and passed. Is the finding disproved?", "expected_output": "Retains an unknown finding because the triggering request order was not exercised.", "files": [], "assertions": [ "Does not withdraw the race finding based on the single-request run", "Names the missing ordering condition", "Proposes a controlled overlapping-request probe" ] } ], "routing": { "should_trigger": [ "Verify these audit findings in the browser before I open the PR.", "Prove these are real. The audit flagged six things and I don't believe half of them.", "Run the probes on /checkout at mobile width.", "Did the fix actually work? Re-check the layout shift on the dashboard.", "Capture both themes on the settings page so I can see the dark-mode contrast.", "Check the contrast on this page for real, not from the hex values.", "The audit says the modal doesn't restore focus. Reproduce it." ], "near_miss": [ "Audit this component for UX bugs. (ui-design: reading source is not this skill)", "Is this ready to ship? (ui-design: the verdict is the audit's, this skill never issues one)", "Write a Playwright e2e test for the checkout flow. (a durable test suite is the repo's, not a one-off probe)", "Set up Chromatic for visual regression. (pixel baselines are Chromatic's)", "Make the close button bigger. (ui-design Build; this skill measures, it does not restyle)", "What's our LCP in production? (field performance is RUM, not a lab probe)" ] } }
-
-
probes
-
axe-scan.md 4.1 KB
# Probe: axe scan Injects axe-core into the loaded page and collects WCAG violations per route and per theme. This is the probe that finally computes contrast, which the whole rules corpus defers on because contrast is a property of two rendered colours and not of a hex value in a class string. ## What it measures Every axe rule in the WCAG 2.0, 2.1 and 2.2 A and AA tag sets, against the DOM as it actually rendered: after hydration, after the theme applied, with real computed colours including whatever a background image or a translucent overlay contributed. Run it once per theme. A palette that passes in light and fails in dark is the single most common contrast defect, and a single-theme scan reports it as clean. ## Recipe ```js import AxeBuilder from '@axe-core/playwright'; const results = await new AxeBuilder({ page }) .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa']) .exclude('iframe[src*="stripe"], iframe[src*="youtube"], [data-consent-banner]') .analyze(); ``` Without the Playwright integration, load `axe.min.js` into the page and call `axe.run(document, { runOnly: { type: 'tag', values: [...] } })`. The results object is identical. Wait for the page to settle first: a scan fired before hydration reports missing labels on controls that are about to receive them. Wait for the route's primary content selector, not a fixed timeout. ## Mapping violations to rule ids Emit each violation under the ui-design rule it corresponds to, so the finding joins the existing corpus rather than arriving as a parallel report: | axe violation id | Rule id | |---|---| | `color-contrast`, `color-contrast-enhanced` | no static rule; emit as `axe:color-contrast` with the theme in `observed` | | `image-alt`, `role-img-alt`, `input-image-alt` | `a11y-image-alt-text` | | `button-name`, `link-name`, `aria-command-name` | `a11y-icon-controls-labeled` | | `label`, `form-field-multiple-labels`, `select-name` | `forms-labels-and-autocomplete` | | `html-has-lang`, `html-lang-valid`, `valid-lang` | `a11y-document-language` | | `th-has-data-cells`, `td-headers-attr`, `table-fake-caption` | `a11y-data-table-semantics` | | `heading-order`, `region`, `landmark-one-main`, `bypass` | `a11y-skip-link-heading-order` | | `video-caption`, `audio-caption` | `a11y-media-captions` | | `aria-*` structural rules, `nested-interactive`, `list` | `a11y-semantic-html-first` | Anything not in the table keeps its axe id under `axe:<id>`. Do not invent a rule id to make a violation look like corpus output. ## What axe does not settle - **Colour as the only carrier of meaning.** axe computes contrast, not semantics: a red border and a green border both pass contrast while being indistinguishable to a viewer who cannot tell them apart. For `a11y-color-only-meaning`, check that each status element carries a non-colour differentiator (an icon, a glyph, or a word) inside its accessible name, and capture the page with `filter: grayscale(1)` on the root as evidence for a human. The grayscale capture is evidence, not a verdict. - **Whether the accessible name is any good.** `button-name` passes on `aria-label="button"`. Read the names the scan returns and flag the useless ones. - **Focus order and restoration.** axe checks landmarks and tab indexes, never where focus went after an action. That is `probes/focus-walk.md`. ## False positives to guard - **Third-party iframes.** Payment fields, embedded players and consent banners produce violations nobody in this repo can fix. Exclude by frame and say so in the report. - **Colour contrast on text over an image or a gradient** returns `incomplete`, not `violation`, because axe cannot sample the backdrop. Treat `incomplete` as its own bucket: report the elements and capture them, never fold them into the violation count. - **Elements hidden behind a closed disclosure** are in the DOM and scanned. Scan the open state deliberately rather than counting the closed one. ## Evidence to write The raw axe results JSON per route and theme (`axe-<route>-<theme>.json`), the violation count by impact, and a capture of the theme that failed. Keep `incomplete` entries in the file; they are the queue for a human. -
console-network.md 3.9 KB
# Probe: console and network Collects everything the browser already told the developer and nobody read. Cheap enough to run on every route in the session, and it regularly produces the most serious finding of a run. ## What it measures Console errors and warnings, uncaught page errors, failed requests, and responses at 400 and above, each tied to the route that produced it. ## Recipe Attach the listeners before the first navigation: ```js const log = { console: [], pageErrors: [], failed: [], badStatus: [] }; page.on('console', (m) => { if (m.type() === 'error' || m.type() === 'warning') log.console.push({ type: m.type(), text: m.text(), at: m.location() }); }); page.on('pageerror', (e) => log.pageErrors.push({ message: e.message, stack: e.stack })); page.on('requestfailed', (r) => log.failed.push({ url: r.url(), method: r.method(), reason: r.failure()?.errorText })); page.on('response', (r) => { if (r.status() >= 400) log.badStatus.push({ url: r.url(), status: r.status() }); }); ``` Exercise the route before reading the log: navigate, wait for the primary content, scroll to the bottom to trigger lazy work, and click the primary action if it is non-destructive. Errors thrown on interaction outnumber errors thrown on load. ## Signatures worth naming | Signature | What it means | |---|---| | `Hydration failed`, `Text content does not match server-rendered HTML` | SSR and client render diverged. On the primary route this is a ship blocker in `ui-design`'s tiering, and it is one of the few defects with no static tell | | `Each child in a list should have a unique "key"` | Reconciliation will reuse the wrong nodes; commonly the cause of a form losing input on re-render | | `Cannot update a component while rendering a different component` | A render-phase state update, usually a loop about to happen under load | | `Warning: validateDOMNesting` | Invalid markup the browser silently restructured; the DOM the code assumes is not the DOM that exists | | `Refused to load ... Content Security Policy` | A resource silently dropped in production and not in dev | | 404 on a font, image, or chunk | Visible as a fallback font or a broken image, and invisible in source | | `ResizeObserver loop completed with undelivered notifications` | Noise in most apps. Note it and move on | ## Reading the result Emit under `runtime:<signature>` where no ui-design rule owns the class, because these are browser findings and the corpus should not be made to look as though it predicted them. Two exceptions map cleanly: | Observation | Rule id | |---|---| | A failed request whose failure renders no user-visible state | `states-no-error-state`, confirmed by `probes/failure-injection.md` | | A 404 on an image referenced by the page | `a11y-image-alt-text` only if the alt text is also missing; otherwise a runtime finding | Report the route, the count, and the first occurrence with its location. A repeated warning firing on every row of a list is one finding with a count. ## False positives to guard - **Dev-only warnings.** React logs `key` warnings, `act` warnings, and double-invoked effects under StrictMode in development and not in production. Run this probe in both modes when it matters, and label which build produced each entry. - **Extension and devtools noise.** A clean context with no extensions avoids most of it. Anything sourced from a `chrome-extension://` URL is not the app's. - **Analytics and consent scripts** blocked by the environment produce failed requests that are correct in a sandbox. Filter by origin and say which origins were filtered. - **The probe's own interceptions.** Requests aborted by another probe's `page.route` show up as failures. Run this probe on an uninjected pass. - **`favicon.ico` 404s.** Real, trivial, and never worth a finding on its own. ## Evidence to write `console-<route>.json` with all four arrays intact, including the entries judged noise. The filtered-out list is what lets a reader disagree with the filter. -
failure-injection.md 6.3 KB
# Probe: failure injection Intercepts the route's data request and returns each failure the happy path never sees. This is the highest-yield probe in the set, because "happy path only" is the most common production UX bug and the one a source read is least able to settle: an error branch that exists in the code says nothing about what renders when it runs. ## What it measures Four injected conditions, each with its own assertions: | Injection | Asks | |---|---| | `500` with a JSON error body | Is there an error state, is its copy usable, does the retry control actually refetch | | `200` with `[]` or a zero-count payload | Is there an empty state, and does it offer a way out | | `200` with a malformed body | Does the parse failure surface, or does the route go blank | | `abort` / offline | Does an offline failure read differently from a server failure | ## Recipe ```js await page.route('**/api/invoices*', (route) => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ error: 'internal', detail: 'ECONNREFUSED 10.0.0.4:5432' }) })); await page.goto(url); await page.waitForLoadState('networkidle'); const rendered = await page.evaluate(() => ({ alerts: [...document.querySelectorAll('[role=alert],[role=status]')].map((n) => n.innerText), bodyText: document.body.innerText.slice(0, 4000), actions: [...document.querySelectorAll('button,a[href]')].map((n) => n.innerText.trim()), blank: document.body.innerText.trim().length < 40, })); ``` Match on the specific path, never a host substring: once analytics and telemetry share an origin, a broad pattern intercepts requests the probe did not mean to break, and the failure it then reports belongs to the harness. Assert the interception fired at all. A route pattern that never matched injects nothing, the page renders its happy path, and the probe reports a passing error state that was never tested. Count the matched requests and treat zero as `unknown`. ## The retry assertion A retry button wired to nothing is identical in source to one that works, and this is the only way to tell them apart. After the error state renders: remove the interception, click the retry control, and assert a new request went out and the content rendered. ```js await page.unroute('**/api/invoices*'); const req = page.waitForRequest('**/api/invoices*', { timeout: 5000 }); await page.getByRole('button', { name: /try again|retry|reload/i }).click(); await req; // throws if the control issues no request ``` A retry that reloads the whole document rather than refetching is a pass with a note, not a fail. ## Reading the result | Observation | Rule id | |---|---| | `blank: true`, or the route's shell disappeared | `states-no-error-state`; if a parent boundary should have caught it, `async-no-error-boundary` | | No `role=alert` and no error copy in `bodyText` | `states-no-error-state` | | Copy matches the leak signature below | `microcopy-leaked-error-message` | | Copy matches the vague set below | `microcopy-vague-error` | | Retry control absent, or present and issuing no request | `states-no-error-state` | | Empty payload renders a zero-row table or a bare "No data" with no action | `states-no-empty-state` | | Offline and server failure render identical copy | `microcopy-vague-error`, low severity | Leak signature: a stack frame (`at fn (file:12:5)`), a typed error name (`TypeError:`, `PrismaClientKnownRequestError`), a driver or dialect token (`ECONNREFUSED`, `SQLSTATE`, `PG::`), a hostname, or an internal IP. The recipe above plants one deliberately, so a probe that does not flag it has an assertion bug. Vague set: `Something went wrong`, `An error occurred`, `Error`, `Invalid`, `Failed`, `Oops`, `Unknown error`, alone and with no cause or next step. ## Mutation failures Point the same interception at the write request rather than the read, and three more rules fall out of one run. Fill the form, submit, and hold the response: - While it is held, the submit control must be disabled and a second click must issue no second request (`forms-no-disable-while-submitting`). A pending state that never appears is the `useFormStatus` bug (`forms-use-form-status-misuse`). - Release it as a 422 with field errors: every field the user typed must still hold its value (`forms-lost-data-on-error`), the errors must be associated with their fields rather than floating in a toast (`forms-error-association`), and focus must move to the first invalid one (`forms-inline-errors-first-focus`). - Release it as a 500 after an optimistic update: the optimistic row must disappear and the previous state must return (`async-optimistic-without-rollback`). An optimistic row that survives a rejected write is a user believing something happened that did not. ## Out-of-order responses The same interception mechanism reproduces `async-out-of-order-responses`, which no other technique reaches. Hold the first query's response, let the second resolve, then release the first, and assert the rendered list matches the second query: ```js let n = 0; await page.route('**/api/search*', async (route) => { const wait = ++n === 1 ? 1500 : 50; // first response arrives last await new Promise((r) => setTimeout(r, wait)); route.continue(); }); await page.getByRole('searchbox').type('ab', { delay: 40 }); ``` Stale results winning is `reproduced`. An AbortController or a `useDeferredValue` guard shows up as the first response never rendering. ## False positives to guard - **A retry that fires on a timer** issues its request without a click. Watch for a request between the error render and the click, and attribute the pass correctly. - **A global error toast from a previous probe** persists across navigations in some apps. Start each injection from a fresh context. - **Service workers and caches** serve the happy path straight past the interception. Clear storage between injections, and disable the service worker in the context. - **Copy assembled from an i18n bundle** that failed to load reads as a missing error state when the real defect is a missing translation. Check the bundle request succeeded. ## Evidence to write One capture per injection, `injection-<condition>.json` holding the rendered text, the alerts, the action labels, and the retry request result. The 500 and the empty capture side by side are usually the whole argument. -
focus-walk.md 5.4 KB
# Probe: focus walk Presses Tab repeatedly and records where focus actually went. Focus bugs are invisible to everyone driving with a mouse, which is why they ship, and they are the class of defect a static read is worst at: `focus-not-restored` depends on what a component library does on unmount, not on what the calling code says. ## What it measures Four things, from one traversal plus one dialog cycle: 1. The focus order, as a list of elements with their boxes, compared against DOM order. 2. Whether each focused element shows a visible indicator, decided by pixel delta. 3. Whether an open dialog holds focus and closes on Escape. 4. Where focus lands after the dialog closes. ## Recipe: the traversal ```js await page.evaluate(() => document.body.focus()); const trail = []; const limit = await page.evaluate(() => document.querySelectorAll('a[href],button,input,select,textarea,[tabindex]').length * 3 + 20); for (let i = 0; i < limit; i++) { await page.keyboard.press('Tab'); const step = await page.evaluate(() => { let el = document.activeElement; while (el?.shadowRoot?.activeElement) el = el.shadowRoot.activeElement; if (!el || el === document.body) return { escaped: true }; const r = el.getBoundingClientRect(); return { tag: el.tagName, label: (el.innerText || el.getAttribute('aria-label') || '').slice(0, 40), box: { x: Math.round(r.x), y: Math.round(r.y), w: Math.round(r.width), h: Math.round(r.height) }, inViewport: r.top >= 0 && r.bottom <= innerHeight, }; }); trail.push(step); if (trail.length > 2 && sameAs(step, trail[0])) break; // cycled back to the start } ``` `sameAs` compares the recorded tag, label and box; the traversal stops when focus returns to where it started, and the cap catches the case where it never does. Piercing shadow roots matters: a design system built on web components reports `document.activeElement` as the host for every step, and the traversal looks like one element repeating. ## Recipe: the focus indicator Computed style cannot decide this. `outline: none` replaced by a `box-shadow` ring is a pass; a ring the same colour as the surface behind it is a fail, and both read identically in CSS. Compare pixels instead: 1. Screenshot the element's box padded by 6px while it is not focused. 2. Focus it with the keyboard, not `el.focus()`, so `:focus-visible` applies. 3. Screenshot the same region. 4. Fail when fewer than roughly 2% of pixels changed. The keyboard detail is load-bearing. Programmatic focus does not match `:focus-visible` in Chromium, so a scripted `el.focus()` reports a missing ring on a control that has one. ## Recipe: the dialog cycle Click the trigger, then assert in order: focus moved inside the dialog subtree; Tab from the last focusable element returns to the first rather than escaping to the page behind; Escape closes it; `document.activeElement` after close is the trigger element itself, not `body` and not the top of the document. Record the trigger's selector before opening, since the element identity is what the restoration assertion needs. ## Paste and IME Two assertions to run while the keyboard is already driving the page, since both are input handlers that look correct in source and fail at runtime. Paste into every text field with `page.keyboard.press('Control+V')` after seeding the clipboard, then read the value back. A field that blocks paste (an `onPaste` preventing default, common on "confirm email" and card-number inputs) ends up empty, which is `forms-dont-block-paste-ime`. For composition, dispatch `compositionstart`, `compositionupdate` and `compositionend` around a multi-character insertion and assert the committed value survived. A field that reformats or validates on every keystroke destroys an in-flight composition, so a Japanese or Korean user cannot type into it at all. ## Reading the result | Observation | Rule id | |---|---| | `escaped: true` mid-traversal, before the cycle completed | `interaction-keyboard-operable` | | Focus order diverges from visual reading order | `interaction-keyboard-operable` | | Pixel delta under threshold on a focused control | `interaction-focus-visible` | | Tab leaves an open modal, or Escape does not close it | `focus-broken-focus-trap` | | Focus after close is not the trigger | `focus-not-restored` | | New content rendered and focus stayed where it was | `focus-on-dynamic-content` | | Focused element outside the viewport with no scroll | `interaction-focus-visible` | | Pasted value did not land, or a composition was destroyed | `forms-dont-block-paste-ime` | ## False positives to guard - **A skip link is invisible until focused and then appears at the top.** It reads as a focus-order anomaly on the first press and is correct. - **An infinite or virtualised list** never cycles back within the limit. Cap the traversal and report the cap rather than a trap. - **Custom widgets that use roving tabindex** (a toolbar, a listbox, a grid) intentionally expose one tab stop and move within it using arrow keys. Fewer tab stops than interactive elements is correct there; test the arrows before reporting an operability failure. - **A dialog that intentionally returns focus elsewhere** after a destructive action (the trigger no longer exists) is correct. Fail only when focus went to `body` or the document top. ## Evidence to write `focus-trail.json` with the full ordered list, the before and after crops for any indicator failure, and the pre-open and post-close `activeElement` for the dialog cycle. -
layout-shift.md 4.9 KB
# Probe: layout shift Holds the data response open long enough for the loading state to render, then measures what moved when it resolved. `states-layout-shift` is a delta between two rendered boxes, so the static greps produce candidates and this probe produces the verdict. Two candidates the greps get wrong in both directions: a skeleton that declares `h-14` still shifts when the loaded row settles at 68px, and a skeleton with no declared height does not shift at all when its parent already reserves the space. ## What it measures Attributed `layout-shift` entries over the window between navigation and data arrival, plus the before and after height of each container that held a placeholder. ## Recipe Register the observer before any script runs, or the entries for the first paint are already gone: ```js await page.addInitScript(() => { window.__shifts = []; new PerformanceObserver((list) => { for (const e of list.getEntries()) { if (e.hadRecentInput) continue; window.__shifts.push({ value: e.value, startTime: e.startTime, sources: e.sources.map((s) => ({ node: s.node?.tagName + (s.node?.id ? '#' + s.node.id : ''), from: s.previousRect, to: s.currentRect, })), }); } }).observe({ type: 'layout-shift', buffered: true }); }); ``` Delay the data, not the whole network. Throttling the shell slows the bundle and moves the shift into a window that has nothing to do with the defect: ```js let release; const gate = new Promise(resolve => { release = resolve; }); let markIntercepted; const intercepted = new Promise(resolve => { markIntercepted = resolve; }); const pattern = '**/api/invoices*'; const delayInvoices = async (route) => { markIntercepted(); await gate; await route.continue(); }; await page.route(pattern, delayInvoices, { times: 1 }); let before, after, shifts; try { await page.goto(url, { waitUntil: 'domcontentloaded' }); await intercepted; // Bound this with the runner timeout; a missed route is unknown. await page.waitForSelector('[data-testid=invoice-skeleton]'); before = await page.locator('#invoice-list').boundingBox(); release(); await page.waitForSelector('[data-testid=invoice-row]'); await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))); after = await page.locator('#invoice-list').boundingBox(); shifts = await page.evaluate(() => window.__shifts); } finally { release(); await page.unroute(pattern, delayInvoices); } ``` Where the app has no test ids, wait on the loaded content's own selector and take the before snapshot immediately after `goto` resolves. Run against a production build. In dev the bundler compiles the route on first navigation, and the resulting paint sequence is an artefact of the dev server. ## Reading the result | Observation | Result | |---|---| | Container height changes on data arrival AND a `layout-shift` entry names it | `reproduced`, fail. Report both heights, the delta in px, and the sum of captured shift values (not the session-window CLS metric) | | Heights differ but no shift entry names the container | Inconclusive from attribution alone; inspect displaced siblings and the capture before deciding | | Shift entries exist but all name elements below the fold that nobody had scrolled to | Record the observation and viewport; tiering belongs to ui-design, not this probe | | Loading and loaded states both observed, no container or sibling movement, and no attributable shift entries | `not-reproduced` for this trigger and viewport | The same probe covers `perf-image-dimensions-and-priority`: an `<img>` with no intrinsic dimensions shows up as a shift source naming the image, with `from` height 0. ## False positives to guard - **Shifts with `hadRecentInput`** fall within the recent-input exclusion window. They are excluded from this observer, but can still be unwanted movement. Inspect them separately if the defect follows an interaction. - **`content-visibility: auto` subtrees** legitimately shift within themselves as they come into view. - **Font swap** produces a real shift that the loading state did not cause. Attribute it: the source node will be a text container, not the placeholder, and the fix belongs to font loading rather than the skeleton. - **The harness itself.** A devtools overlay, an injected banner, or a screenshot-time scroll all generate entries. Compare the entry timestamps against the injected delay window and drop anything outside it. - **A run with no loading state observed at all** means the delay never applied or the data was cached. Assert the skeleton was seen; if it was not, the result is `unknown`, not a pass. ## Evidence to write `shifts.json` with the entries and their sources, the before and after `boundingBox` per container, and two captures: one with the loading state up, one immediately after resolution. The pair is what makes the finding obvious to a reader who will not read the numbers. -
target-size.md 5.8 KB
# Probe: target size Measures the real hit area of every visible interactive element at a touch viewport, so `interaction-target-size` stops being a guess about class names. The static rule greps for small sizing classes and cannot see three things that decide the finding: a pseudo-element that expands the hit area, a transparent overlay that owns the click, and padding on an ancestor that is the actual target. All three are common in shipped component libraries, and all three make the static hit a false positive. ## What it measures Per element: the bounding box, and the *effective* hit area found by hit-testing. An element passes when a 44x44 CSS px box centred on it resolves to that element or one of its descendants at the sampled grid points. This is a hit-test estimate, not proof of every point in the area. ## Recipe Run at a touch viewport (360x800, `hasTouch: true`, `isMobile: true`). The 44px floor is a touch threshold; measuring it under a fine pointer reports desktop-dense UI as broken. ```js const results = await page.evaluate(() => { const SEL = 'a[href], button, input:not([type=hidden]), select, textarea, summary,' + '[role=button], [role=link], [role=checkbox], [role=radio], [role=switch],' + '[role=tab], [role=menuitem], [role=option], [tabindex]:not([tabindex="-1"])'; const cssPath = (node) => { const parts = []; while (node && node.nodeType === 1) { if (node.id) { parts.unshift('#' + CSS.escape(node.id)); break; } const tag = node.localName; const siblings = node.parentElement ? [...node.parentElement.children].filter(n => n.localName === tag) : [node]; parts.unshift(`${tag}:nth-of-type(${siblings.indexOf(node) + 1})`); node = node.parentElement; } return parts.join(' > '); }; const out = []; for (const el of document.querySelectorAll(SEL)) { if (!el.checkVisibility({ checkOpacity: true, checkVisibilityCSS: true })) continue; const r = el.getBoundingClientRect(); if (r.width === 0 || r.height === 0) continue; const cx = r.left + r.width / 2, cy = r.top + r.height / 2; if (cx < 0 || cy < 0 || cx >= document.documentElement.clientWidth || cy >= document.documentElement.clientHeight) continue; if (getComputedStyle(el).pointerEvents === 'none') continue; // Sample just inside the edges: hit testing at an exact boundary can select a neighbor. const offsets = [-21.5, -11, 0, 11, 21.5]; const points = offsets.flatMap(dx => offsets.map(dy => [dx, dy])); const covered = points.every(([dx, dy]) => { const hit = document.elementFromPoint(cx + dx, cy + dy); return hit && (hit === el || el.contains(hit)); }); out.push({ selector: cssPath(el), text: (el.innerText || el.getAttribute('aria-label') || '').slice(0, 40), box: { w: Math.round(r.width), h: Math.round(r.height) }, effective44: covered, inProse: el.matches('a[href]') && getComputedStyle(el).display === 'inline' && !!el.closest('p, [class*=prose]'), }); } return out; }); ``` The selector helper lives inside `evaluate`, since page execution cannot access functions defined in the driver. Confirm ancestor event delegation separately; a hit on a generic parent does not prove it activates the child. ## Reading the result | Condition | Result | |---|---| | `box` under 44 on either axis AND `effective44` false | `reproduced`, fail | | `box` under 44 but `effective44` true | `not-reproduced`; the hit area is expanded. Record both numbers | | `box` between 44 and 47 | pass with a note. 48 is the build target, 44 is the conformance floor | | `inProse` true | Inspect the inline-link exception; do not exempt buttons merely because they sit in a list or table | Report `observed: { width, height, effectiveHitArea }` and the viewport. A finding without the viewport is not reproducible, because the same control legitimately measures differently under `pointer: fine`. ## Hover-only affordances The same enumeration settles `mobile-hover-only-affordance`, because both questions are "can a touch user reach this control". Record the visible interactive set at the touch viewport, then dispatch a synthetic `pointerover` and `mouseover` over each container and record it again. Treat differences as candidates. Synthetic events do not reproduce CSS `:hover` or a real tap sequence; confirm through the supported driver's pointer and touch actions before reporting a touch failure. The static rule catches `group-hover:opacity-100` and its relatives. What it cannot see is a control revealed by JavaScript on hover, or one revealed on hover and also on focus, which is the correct pattern and reads identically to the broken one until the page runs. ## False positives to guard - **Corner hit-testing catches its own overlay.** A full-page modal backdrop or a sticky header sitting over the control returns the overlay at every corner, so `effective44` reads false for a control that is fine. Scroll the element into view and dismiss transient chrome before measuring. - **Elements inside a closed menu, drawer, or tab panel** are visible to the selector but not to the user. `checkVisibility` handles `display: none` and `visibility: hidden`, not a panel positioned off-canvas: filter boxes whose centre falls outside the viewport. - **Controls that are decoration.** `pointer-events: none` elements match the role selectors but take no input. Check the computed value before measuring. - **A repeated component reports once per instance.** Twenty table rows with the same undersized icon button is one finding with a count, not twenty findings. ## Evidence to write `target-size.json` with the full element list, and one annotated screenshot per failing element (the viewport capture is enough; a crop around the control is better). Keep the passing rows in the JSON: they are what makes a `not-reproduced` result checkable. -
theme-locale-matrix.md 4.8 KB
# Probe: theme and locale matrix Captures the same route across viewport, theme, string length, and text direction. `dark-i18n-untested` and `dark-i18n-rtl-untested` default to backlog precisely because nobody has looked; this probe is the looking. ## What it measures The matrix is viewport x theme, with two extra passes layered on the widest and narrowest widths: | Axis | Values | |---|---| | Viewport | 360x800 with touch, 1280x800 | | Theme | light, dark | | Locale | source strings, pseudo-locale (expanded) | | Direction | `ltr`, `rtl` | Eight captures for the base matrix is the useful default. The full cross product is sixteen and mostly redundant: run the pseudo-locale and RTL passes at one width each unless the base matrix already showed an edge. ## Setting the theme, correctly ```js await page.emulateMedia({ colorScheme: 'dark' }); ``` This is the whole fix only for an app themed purely by `prefers-color-scheme`. Most Tailwind apps toggle a `class="dark"` or `data-theme` attribute on `<html>`, and for those the media emulation changes nothing while the probe reports a clean dark pass it never took. Drive the app's own control, then assert: ```js const applied = await page.evaluate(() => document.documentElement.className + '|' + (document.documentElement.dataset.theme ?? '')); ``` If neither the class nor the attribute changed, the theme did not apply and the capture is `unknown`, not a dark-mode pass. Where the app persists the choice, setting `localStorage` before the first navigation is more reliable than clicking a toggle that may be inside a menu. Run `probes/axe-scan.md` at each theme. Contrast is the finding this matrix exists to produce, and it is computed there. ## Pseudo-locale Where the app has an i18n layer, switch to a pseudo-locale it supports, or intercept the message bundle and transform the values: accent the letters so untranslated strings stand out, and pad to roughly 140% of the source length so the layout meets the German and Finnish case. Where it has no i18n layer, expand in the DOM as `probes/viewport-stress.md` does, and say in the report that the expansion was applied post-render. The two are not equivalent: a DOM pass cannot reach text drawn into a canvas, placeholder attributes, or strings a later render replaces. What the pass is looking for is layout failure, not prose: buttons that grow past their container, labels that clip, nav items that wrap into two rows and push the header taller, tabs that overflow with no scroll affordance. ## RTL ```js await page.evaluate(() => { document.documentElement.dir = 'rtl'; }); ``` Then re-run the overflow check from `probes/viewport-stress.md` and capture. Physical properties are what break: `margin-left`, `left`, `text-align: left`, `border-l`, and a `translateX` that assumes one direction. They show up as elements crowding the wrong edge, icons on the wrong side of their label, and drawers sliding in from the wrong side. Setting `dir` on the document does not translate anything, so read the capture for geometry only. A layout that mirrors cleanly passes even with the source strings still in English. ## Reading the result | Observation | Rule id | |---|---| | Theme attribute never changed | `unknown`, reason `theme-not-applied`. Never a pass | | axe `color-contrast` violations present in dark and absent in light | `dark-i18n-untested`, `reproduced` | | Hardcoded light surface visible in the dark capture (a white card, a black-on-dark icon) | `dark-i18n-untested` | | Overflow or clipping under the pseudo-locale that is absent at source length | `dark-i18n-untested` at the layout end; report the element | | Elements crowding the wrong edge under `dir=rtl` | `dark-i18n-rtl-untested` | An app with no dark theme at all is not a finding here. Report it as not applicable and move on; whether the product should have one is a `product-design` question. ## False positives to guard - **A capture taken before the theme transition finished** shows a half-swapped page. Wait for the transition, or disable transitions for the capture. - **Images and illustrations authored for one theme** are a real finding but not a CSS one; the fix is a second asset, so report it with the capture rather than as a token defect. - **RTL on a page of source-language prose** legitimately left-aligns paragraphs in some designs. Judge chrome and controls, not body copy. - **Pseudo-locale expansion inside a fixed-width design that was never meant to localise.** Check whether the product ships other locales before reporting expansion failures at full severity. ## Evidence to write `<route>-<width>-<theme>.png` for the base matrix, plus `<route>-pseudo.png` and `<route>-rtl.png`. Keep the naming mechanical: this probe's output is compared across runs more than any other, and a matrix whose filenames drift cannot be diffed. -
viewport-stress.md 5.3 KB
# Probe: viewport stress Narrows the viewport and lengthens the content, then finds what overflowed. `layout-long-content-safety` asks whether a layout survives a long name, a dense table, and a small screen; none of those are answerable from a class string. ## What it measures At each width: whether the document scrolls horizontally, which element caused it, and which text is clipped without an affordance. Then the same checks again with every text node tripled in length. 320px is not an arbitrary floor. WCAG 1.4.10 requires reflow at 320 CSS px, which is what a 1280px viewport at 400% zoom becomes, so a clean pass at 320 is the reflow criterion met. Test the width; do not try to drive browser zoom. ## Recipe: overflow ```js for (const width of [320, 360, 768, 1280]) { await page.setViewportSize({ width, height: 800 }); const overflow = await page.evaluate(() => { const doc = document.scrollingElement; if (doc.scrollWidth <= doc.clientWidth + 1) return null; const culprits = []; for (const el of document.querySelectorAll('body *')) { const r = el.getBoundingClientRect(); if (r.width === 0 || r.height === 0) continue; if (getComputedStyle(el).position === 'fixed') continue; if (r.right > innerWidth + 1 || r.left < -1) { culprits.push({ tag: el.tagName, cls: el.className?.toString().slice(0, 60), right: Math.round(r.right), width: Math.round(r.width) }); } } return { scrollWidth: doc.scrollWidth, clientWidth: doc.clientWidth, culprits }; }); } ``` The culprit list nests: a wide child reports its ancestors too. Take the deepest element in the list as the cause, and report one finding per distinct cause rather than one per element in the chain. ## Recipe: clipped text ```js const clipped = await page.evaluate(() => [...document.querySelectorAll('body *')].filter((el) => { if (!el.firstChild || el.firstChild.nodeType !== Node.TEXT_NODE) return false; const s = getComputedStyle(el); const cut = el.scrollWidth > el.clientWidth + 1 || el.scrollHeight > el.clientHeight + 1; const hidden = s.overflow === 'hidden' || s.overflowX === 'hidden' || s.overflowY === 'hidden'; const affordance = s.textOverflow === 'ellipsis' || s.webkitLineClamp !== 'none'; return cut && hidden && !affordance; }).map((el) => ({ text: el.innerText.slice(0, 60), cls: el.className?.toString().slice(0, 60) }))); ``` Text cut off with no ellipsis and no line clamp is invisible truncation: the user cannot tell there was more. With an ellipsis it is a deliberate pattern, and the finding is only whether the full value is reachable (a title attribute, a tooltip, a details view). ## Recipe: long content Triple every text node in place and re-run both checks. This is the mechanical form of "does this survive a real customer name": ```js await page.evaluate(() => { const w = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT); const nodes = []; while (w.nextNode()) nodes.push(w.currentNode); for (const n of nodes) if (n.nodeValue.trim()) n.nodeValue = n.nodeValue.trim().repeat(3); }); ``` Mutating the DOM directly outlives one render in most apps, but a re-render restores the original strings. Where the app re-renders on an interval or a subscription, use the pseudo-locale route in `probes/theme-locale-matrix.md` instead, which expands the strings at their source. ## Reading the result | Observation | Rule id | |---|---| | Horizontal document scroll at any width | `layout-long-content-safety`, `reproduced` | | Clipped text with no ellipsis or clamp | `layout-long-content-safety` | | Overflow appears only after tripling | `layout-long-content-safety`, at lower severity: real but content-dependent | | Page scales rather than reflows, or pinch zoom is blocked | `mobile-viewport-scaling` | | Body copy computing under 16px at a mobile width | `type-readable-scale`; on an input, `forms-mobile-input-font-size`, since iOS zooms the page on focus below 16px | ## False positives to guard - **Deliberate horizontal scrollers.** A carousel, a wide data table in its own `overflow-x: auto` container, and a code block are correct. The failure is the *document* scrolling, so check whether the culprit sits inside a scroll container before reporting it. - **Off-canvas drawers** parked at `translateX(100%)` extend past the right edge by design. Their computed transform tells you; exclude elements whose parent is `overflow: hidden` and whose offset is a whole viewport width. - **A 1px overflow** is a rounding artefact of fractional layout, not a defect. The `+1` tolerances in the recipes are there for that, and widening them further hides real findings. - **Headless font substitution.** Headless Chromium ships different default fonts than the developer's machine, so text metrics and wrapping differ. Confirm a clipped-text finding against a capture before reporting it, and prefer the repo's own container image when it has one. - **Sticky and fixed chrome** is excluded from the culprit list on purpose, because it is positioned relative to the viewport. Check it separately by scrolling content beneath it. ## Evidence to write `overflow-<width>.json` per width with the culprit chain, plus a full-page capture at every width that failed. The capture is what shows a reader whether the overflow is a stray shadow or half the page. -
web-vitals.md 3.9 KB
# Probe: web vitals Measures LCP, CLS and INP in the lab, with attribution. The value here is naming the element, not scoring the page: `ui-design`'s defer table sends budgets to Lighthouse and field data to RUM, and this probe does not take that over. Read that boundary literally. A single headless run on a developer machine is not a performance verdict, and reporting it as one is how a team learns to distrust the whole report. ## What it measures | Metric | What the probe adds | |---|---| | LCP | Which element is the largest contentful paint, and which of its four phases dominates | | CLS | The same entries as `probes/layout-shift.md`, summed over the page load rather than one data window | | INP | The response time of a specific scripted interaction, not a page score | ## Recipe Inject the attribution build before navigation so the observers register early: ```js await page.addInitScript({ path: require.resolve('web-vitals/dist/web-vitals.attribution.iife.js') }); await page.addInitScript(() => { window.__vitals = []; addEventListener('load', () => { const push = (m) => window.__vitals.push({ name: m.name, value: m.value, rating: m.rating, attribution: m.attribution }); webVitals.onLCP(push); webVitals.onCLS(push); webVitals.onINP(push); }); }); ``` Then drive a real interaction so INP has something to measure. INP with no interaction reports nothing, which reads as a pass: ```js await page.getByRole('button', { name: /save|search|submit/i }).click(); await page.waitForTimeout(500); const vitals = await page.evaluate(() => (window.__vitals ??= [])); ``` Run against a production build, on a fresh context with no cache, and with CPU throttled (CDP `Emulation.setCPUThrottlingRate`, 4x) so a fast machine does not hide a slow interaction. Report the throttling factor with every number; without it the numbers mean nothing across machines. ## Reading the result Report attribution, and flag only what attribution makes actionable: | Observation | Rule id | |---|---| | LCP element is an image with no `priority` or `fetchpriority` and a late `resourceLoadDelay` | `perf-image-dimensions-and-priority`, `reproduced` | | LCP element is below the fold, or is a spinner | The route's largest paint is not its content; report as a runtime finding with the element | | CLS above 0.1 with sources naming a placeholder | Defer to `probes/layout-shift.md`, which has the before and after boxes | | LCP dominated by `resourceLoadDelay` on a data request, with the whole route blank until it resolves | `async-no-suspense-boundary`, confirming: nothing streamed, so the slowest fetch gated the first paint | | INP above 200ms with `attribution.longAnimationFrames` naming a script | Runtime finding naming the script. No rule owns it, and none should | Everything else goes in the report as measured context, not as a finding. A LCP of 2.8s on one throttled headless run is a number, not a defect. ## False positives to guard - **Dev builds.** On-demand compilation on first navigation lands squarely in LCP. A dev-mode LCP measures the bundler. - **A cold cache on every run** overstates repeat-visit performance and understates nothing, so it is the right default, but say which it was. - **The first run after a build** pays for cold server-side caches too. Discard the first navigation and measure the second. - **INP with no interaction, or with an interaction that opens a new page,** reports nothing or the wrong thing. Assert an interaction was recorded. - **Headless font substitution** changes text paint timing. It moves LCP by a small amount and is not worth correcting for, but it is worth knowing when a number sits on a threshold. ## Evidence to write `vitals.json` with the raw metric objects including attribution, the throttling settings, the build mode, and which navigation was measured. Attribution is the whole payload; a JSON file holding three numbers and no attribution is not worth writing.
-
-
references
-
evidence-output.md 6.5 KB
# Evidence and Output The delta this skill adds to `ui-design`'s finding document, and nothing else. That schema, in its `references/output-adapters.md`, stays the single owner of the finding object, the three counts, the tiers, and the verdict. Two files describing one JSON shape is how the two drift apart, so what follows adds fields and redefines none. ## Table of contents - [The session block](#the-session-block) - [The verification block](#the-verification-block) - [How a probe result changes a finding](#how-a-probe-result-changes-a-finding) - [Artifacts](#artifacts) - [The clearing re-run](#the-clearing-re-run) - [Terminal rendering](#terminal-rendering) - [Self-check additions](#self-check-additions) ## The session block One per run, alongside `audit`. Without it a reader cannot tell a clean run from a run that never reached the app: ```json { "session": { "driver": "playwright", "browser": "chromium 131", "buildMode": "production", "baseUrl": "http://localhost:3000", "routes": [{ "url": "/invoices", "from": "src/app/invoices/page.tsx", "status": 200 }], "viewports": [{ "width": 360, "height": 800, "touch": true }, { "width": 1280, "height": 800 }], "themes": ["light", "dark"], "auth": "seeded-storage-state", "dataFixture": "seed/invoices-24", "probesRun": ["axe-scan", "target-size", "focus-walk", "failure-injection"], "probesSkipped": [{ "probe": "web-vitals", "reason": "dev build only; no production server available" }], "artifactDir": ".ui-verification/2026-09-04T0912Z" } } ``` `probesSkipped` is required and is not allowed to be inferred from absence. A probe missing from both arrays is a reporting bug. ## The verification block Appended to any finding a probe touched: ```json { "verification": { "probe": "target-size", "route": "/invoices", "viewport": { "width": 360, "height": 800, "touch": true }, "theme": "light", "result": "reproduced", "observed": { "width": 32, "height": 32, "effectiveHitArea": 32 }, "expected": { "min": 44 }, "evidence": [".ui-verification/2026-09-04T0912Z/invoices/target-size/360-light.png", ".ui-verification/2026-09-04T0912Z/invoices/target-size/measurements.json"], "reruns": 2, "reason": null, "clearedBy": null } } ``` | Field | Required when | Description | |---|---|---| | `probe` | always | the probe file's name without `.md` | | `route`, `viewport`, `theme` | always | the exact conditions. A measurement without them is not reproducible | | `result` | always | `reproduced \| not-reproduced \| unknown` | | `observed` | result is `reproduced` | the measurement, in units | | `expected` | when the rule has a threshold | the threshold the measurement is judged against | | `evidence` | result is `reproduced` or `not-reproduced` | artifact paths that exist on disk | | `reruns` | always | how many times the probe ran. A reported fail is at least 2 | | `reason` | result is `unknown` | why the probe could not decide: `auth-required`, `no-route`, `theme-not-applied`, `driver-lacks-interception`, `flaky`, `probe-error` | | `clearedBy` | after a fix | the re-run that proved the finding gone | `observed` here replaces the static read's `observed` on the parent finding. That substitution is the deliverable: `h-8 on line 42` becomes `32x32px at 360px width with touch emulation`. ## How a probe result changes a finding | Probe result | Parent finding | |---|---| | `reproduced` | Stays `fail`. `observed` comes from the measurement. Tier is unchanged: tiering is the audit's | | `not-reproduced` | Withdraw only when the probe exercised the alleged trigger. Record the trigger and measurement in `consideredAndRejected`; otherwise retain an `unknown` finding naming the missing condition | | `unknown` | Stays as a finding with `result: "unknown"` and the probe's `reason`. It never becomes a `pass` | A probe that found something no static rule predicted enters as a new finding against the rule it is primary for. Where no rule owns it, use `axe:<violation-id>` or `runtime:<signature>` as the `rule` value and say in the text that it came from the browser rather than from the corpus. Do not stretch a rule id to make a browser finding look like a predicted one. ## Artifacts ```text .ui-verification/<run-id>/<route-slug>/<probe>/<width>-<theme>.png .ui-verification/<run-id>/<route-slug>/<probe>/<name>.json ``` Mechanical naming, because these get compared across runs more than they get read once. Add the directory to the repo's ignore file if it is not already covered; a verification run that leaves a hundred screenshots in the working tree ends up committed. Every path in an `evidence` array exists on disk when the report is written. A cited artifact that was never saved is worse than no citation: it reads as verified and cannot be checked. ## The clearing re-run A fix is proved by the probe, not by the diff. Re-run with every condition identical: route, viewport, theme, seed, injected failure, build mode. ```json { "clearedBy": { "ranAt": "2026-09-04T09:41:00Z", "result": "not-reproduced", "observed": { "width": 48, "height": 48 }, "evidence": [".ui-verification/2026-09-04T0941Z/invoices/target-size/360-light.png"] } } ``` Keep the before artifact. Writing the after capture over the before is how a report loses its own argument. A re-run that still reproduces is recorded as `clearedBy.result: "reproduced"` on an `applied` finding, which is the evidence `ui-design`'s revert-on-failed-fix rule runs on. ## Terminal rendering Same adapter, two additions: - Each reproduced finding gains one line under its fix line: `Probe: <probe> · <route> · <width>px <theme> · <observed>`. - The footer gains a session line before the defer-to block: driver, build mode, routes, probes run, probes skipped with reasons. - `SHIP VERDICT` is omitted on a standalone run, per SKILL.md. ## Self-check additions Added to the audit's existing codes: ```text "probe-not-run" // a finding claims a probe that is absent from session.probesRun "evidence-missing" // a cited artifact path does not exist on disk "single-run-fail" // a reproduced finding with reruns < 2 "conditions-not-recorded" // a verification block missing route, viewport, or theme "cleared-without-rerun" // applied: true on a probed finding with no clearedBy "unknown-as-pass" // a probe returned unknown and the finding was dropped ``` `unknown-as-pass` is the one that matters. Every other failure mode here makes the report weaker; that one makes it wrong, by converting "we could not check" into "we checked and it was fine". -
rule-coverage.md 7.5 KB
# Rule Coverage Which probe decides which `ui-design` rule. Read this when an audit hands over a list of rule ids: take the probe each id maps to, run those, and pass the rest through untouched. ## Table of contents - [Roles](#roles) - [Coverage table](#coverage-table) - [One-line runtime confirmations](#one-line-runtime-confirmations) - [What no probe decides](#what-no-probe-decides) - [What stays with other tools](#what-stays-with-other-tools) ## Roles | Role | Meaning | |---|---| | **primary** | The browser is where the defect lives. The static check produces candidates at best, and the probe is the verdict | | **confirming** | The static check decides it well. The probe adds computed evidence, catches what the source read could not see, or checks the rendered result of a correct-looking source | | **capture-only** | No probe decides it. The run supplies rendered captures so a human, or the audit's own judgement, can | A primary rule that the probe could not run for stays `unknown`. A confirming rule whose probe could not run keeps its static result, and the report says the confirmation did not happen. ## Coverage table | Rule id | Probe | Role | |---|---|---| | `a11y-color-only-meaning` | axe-scan | confirming | | `a11y-data-table-semantics` | axe-scan | confirming | | `a11y-disabled-control-tooltip` | axe-scan, focus-walk | confirming | | `a11y-document-language` | axe-scan | confirming | | `a11y-icon-controls-labeled` | axe-scan | confirming | | `a11y-image-alt-text` | axe-scan | confirming | | `a11y-media-captions` | axe-scan | confirming | | `a11y-semantic-html-first` | axe-scan | confirming | | `a11y-skip-link-heading-order` | axe-scan, focus-walk | confirming | | `a11y-tooltip-no-interactive` | focus-walk | confirming | | `async-no-error-boundary` | failure-injection | primary | | `async-no-suspense-boundary` | web-vitals | confirming | | `async-optimistic-without-rollback` | failure-injection | primary | | `async-out-of-order-responses` | failure-injection | primary | | `dark-i18n-rtl-untested` | theme-locale-matrix | primary | | `dark-i18n-untested` | theme-locale-matrix, axe-scan | primary | | `focus-broken-focus-trap` | focus-walk | primary | | `focus-not-restored` | focus-walk | primary | | `focus-on-dynamic-content` | focus-walk | primary | | `forms-dont-block-paste-ime` | focus-walk | confirming | | `forms-error-association` | failure-injection, axe-scan | primary | | `forms-inline-errors-first-focus` | failure-injection, focus-walk | primary | | `forms-input-affix-hit-target` | target-size | confirming | | `forms-labels-and-autocomplete` | axe-scan | confirming | | `forms-lost-data-on-error` | failure-injection | primary | | `forms-mobile-input-font-size` | viewport-stress | primary | | `forms-no-disable-while-submitting` | failure-injection | primary | | `forms-use-form-status-misuse` | failure-injection | confirming | | `interaction-focus-visible` | focus-walk | primary | | `interaction-keyboard-operable` | focus-walk | primary | | `interaction-target-size` | target-size | primary | | `layout-long-content-safety` | viewport-stress | primary | | `microcopy-leaked-error-message` | failure-injection | primary | | `microcopy-vague-error` | failure-injection | primary | | `mobile-hover-only-affordance` | target-size | primary | | `mobile-viewport-scaling` | viewport-stress | primary | | `nav-live-region-feedback` | failure-injection, axe-scan | confirming | | `nav-semantic-links` | focus-walk, axe-scan | confirming | | `perf-image-dimensions-and-priority` | layout-shift, web-vitals | primary | | `perf-lazy-load-offscreen` | console-network | confirming | | `perf-virtualize-large-lists` | see below | confirming | | `slop-affordance-mismatch` | theme-locale-matrix | capture-only | | `slop-decoration-no-role` | theme-locale-matrix | capture-only | | `slop-faux-product-chrome` | theme-locale-matrix | capture-only | | `slop-near-duplicate-scale` | theme-locale-matrix | capture-only | | `slop-token-drift` | theme-locale-matrix | capture-only | | `slop-unverifiable-proof` | theme-locale-matrix | capture-only | | `states-layout-shift` | layout-shift | primary | | `states-no-empty-state` | failure-injection | primary | | `states-no-error-state` | failure-injection | primary | | `type-hover-weight-shift` | layout-shift | confirming | | `type-readable-scale` | viewport-stress | primary | Every rule in `ui-design/rules/` has a row. When a rule is added or removed there, this table changes with it; a rule with no row is a rule the browser cannot help with, and that is a decision to record here rather than an omission. The two rules the audit corpus already marks `detect: rendered`, `states-layout-shift` and `layout-long-content-safety`, are both primary here. That is the point of the mapping: a rule whose own file says it needs the browser had, until now, no browser to be run in. ## One-line runtime confirmations Three rules are settled by a single evaluate call rather than a probe file. Run them alongside whichever probe is already on the route: - **`perf-virtualize-large-lists`**: count the rendered rows against the payload length. A thousand records and a thousand DOM nodes is the defect; a thousand records and forty nodes is a working virtualiser. - **`perf-lazy-load-offscreen`**: record image and iframe requests issued before any scroll. An asset below the fold fetched at load is the finding, and `probes/console-network.md` already holds the request log. - **`forms-use-form-status-misuse`**: hold the submit response open. The bug is a pending state that never appears because the status hook is always false in the component that owns the form, and holding the response is what makes its absence visible. ## What no probe decides The six `slop-` rules are aesthetic judgements against a threshold, and a measurement cannot make them. What the browser adds is the evidence they should have been judged on in the first place: `ui-design`'s Deslop scope already requires rendering at desktop and mobile before editing, because compounding slop is a visual property and reading JSX is the wrong evidence for it. This skill supplies those captures. It does not score them. The same holds for the parts of a design no rule encodes: hierarchy, restraint, whether the dark theme looks intentional or merely inverted. Captures go in the report; the judgement stays where it was. ## What stays with other tools Verifying in a browser does not absorb the defer table. These stay out: | Concern | Owner | Why it is not a probe | |---|---|---| | Performance budgets and scoring | Lighthouse CI | A single lab run is attribution, not a budget. `probes/web-vitals.md` says so in its own opening | | Field performance | RUM (Speed Insights, Sentry, Datadog) | No lab run is field data | | Pixel regression against a baseline | Chromatic, Percy | Requires a stored baseline and a review workflow this skill does not own | | Cross-browser behaviour | The project's own test matrix | These probes run one engine. Say which | | End-to-end flow correctness | Playwright or Cypress tests in the repo | A probe is a one-off measurement, not a regression suite. Where a finding deserves a permanent guard, the deliverable is a test, and writing it is the repo's job | | Bundle size, dependency CVEs, type errors, lint | size-limit, Dependabot, tsc, eslint | Unchanged by having a browser | The line worth holding: this skill runs probes and throws the browser away. If a check should run on every commit, it belongs in the repo's test suite, and the right outcome of a reproduced finding is often a fix plus a test, not a standing probe. -
session-setup.md 6.6 KB
# Session Setup Everything that has to be true before the first probe runs. A probe result is only as good as the session under it, and every failure mode here produces a confident wrong answer rather than an error. ## Table of contents - [Driver](#driver) - [Build mode](#build-mode) - [Booting the app](#booting-the-app) - [Auth and seeded data](#auth-and-seeded-data) - [Resolving routes from a diff](#resolving-routes-from-a-diff) - [Determinism](#determinism) - [When to stop](#when-to-stop) ## Driver Use the supported browser surface exposed by the host. The recipes below use Playwright syntax as an adapter; do not assume `evaluate`, interception, or browser launch is available on another driver. **Playwright** is the default. It gives request interception, network and CPU emulation, a fresh isolated context per run, and repeatability. Use the repo's own `@playwright/test` install when there is one, so the browser version matches CI. Where a browser binary is already provisioned in the environment, point Playwright at it with `executablePath` rather than downloading another. **A browser driven over the Chrome DevTools tool family** is the fallback: use it when the app cannot be automated headlessly, when the probe needs a real logged-in profile, or when installing anything in the repo is off the table. The tradeoff decides which probes exist: | Probe | Needs interception or network control | |---|---| | failure-injection | Yes. Does not exist without it | | layout-shift | Yes, to hold the data response | | web-vitals | Yes, for CPU and network throttling | | theme-locale-matrix | Only for the pseudo-locale bundle transform | | axe-scan, target-size, focus-walk, viewport-stress, console-network | No | Under a driver without interception, report those probes as skipped with the reason, never as passed. Silently dropping half the battery is the failure this table exists to prevent. ## Build mode | Probe | Build | |---|---| | web-vitals, layout-shift | Production build, served from the production server | | Everything else | Dev is fine, and faster to iterate | A dev server compiles the route on first navigation. That compilation lands in LCP and can generate paint sequences that exist nowhere in production, so a dev-mode perf number measures the bundler. Development also enables React's extra warnings, which is useful for the console probe and misleading for everything else: label which build produced each result. ## Booting the app Read the manifest's scripts rather than guessing the command. Prefer, in order, an explicit instruction from the user, a documented command in the repo's own agent instructions, then the conventional script (`dev`, `start`, `preview`). Wait for readiness by polling the URL until it answers, never by sleeping. A fixed sleep is either too short, which produces a run against a half-started server, or too long, which is why nobody runs the suite twice. Use a free port and record the process started by this run. Reuse an existing server only after confirming its checkout and build. Stop only a server this run owns; never kill an unrelated listener to claim port 3000. ## Auth and seeded data Reuse a session rather than driving a login form: a saved storage state, a test-account cookie the repo already provisions, or the app's own dev-login route. Never type real credentials, and never read them out of the environment into a probe script. A route that needs auth with no session available returns `unknown` with reason `auth-required`. This is the single most important `unknown` in the skill: an unauthenticated app usually redirects to a login page that renders perfectly, and every probe then passes against the wrong page. Assert the final URL is the route requested before believing any result from it. Data matters as much as auth. An account with no records makes every list render its empty state, so `states-no-empty-state` passes and `states-layout-shift` cannot fire. Note the account's data shape in the session block, and prefer a seeded fixture over whatever the developer's local database happens to hold. ## Resolving routes from a diff Map changed files to URLs before anything else, because a probe run against the wrong route is worse than no run: 1. **A route file is its own URL.** Next.js `app/<segment>/page.tsx` maps to `/<segment>`; `pages/<name>.tsx` likewise. Dynamic segments need a real value: take one from the seeded data, never a placeholder that 404s. 2. **A component file needs its importers.** Walk imports upward until a route file is reached. A component reachable from three routes gets probed on the one the diff touched, or on the most critical of them when the diff is the component itself. 3. **A component with no reachable route** falls back to the repo's Storybook or component workshop if one exists, and to `unknown` with reason `no-route` if not. A component mounted in isolation is a real target for target-size, focus-walk and axe-scan, and a poor one for layout-shift and failure-injection, which need the app's real data layer. 4. **A shared layout or token file** changes every route. Probe the two or three highest-traffic routes rather than all of them, and say which were chosen. Confirm each resolved URL returns 200 and renders the changed component before probing it. A route that resolved but renders a different branch produces findings about code nobody changed. ## Determinism A probe that flips between runs is worse than no probe: it spends the credibility that measurement was supposed to buy. - **Fix the viewport and the device scale.** Never let the probe inherit whatever the window happened to be. - **Disable animation for captures** with reduced motion, and run motion-sensitive checks in a separate pass with it on. Reduced motion is also a code path, and it can be broken. - **Freeze the data.** The same seeded fixture on every run, so a list length change does not read as a layout regression. - **Fresh context per probe.** Storage, service workers, and caches carried between probes are how one probe's injection leaks into the next one's result. - **Repeat uncertain results** when timing or inconsistent evidence warrants it. A deterministic captured failure needs no ritual second run. Record inconsistent results and their conditions as inconclusive. ## When to stop Stop and report the session, not a finding list, when the app will not boot, the resolved route 404s, auth cannot be established, or the driver lacks what the selected probes need. Each of those is a reportable outcome. None of them is a clean run, and a report that presents an empty finding list without the session block is indistinguishable from a passing verification.
-
-
SKILL.md 13.3 KB
--- name: ui-verification description: Runs scoped browser probes for focus, hit targets, overflow, themes, request failures, and performance attribution, with evidence linked to UI rule IDs. Use when asked to "verify this in the browser", "reproduce this finding", or "check the fix". For source audits and severity use ui-design; field metrics require RUM or CrUX. compatibility: Requires access to the target app and browser automation. Bundled JavaScript recipes use the Playwright page API. --- # UI Verification Owns the browser session. Every other UI skill in this repo reasons about source and infers what the user will see; this one loads the page and measures it. - **IS:** booting the app, driving it with a browser, and running the probes that decide a rule at runtime: computed boxes, injected failures, observed layout shift, a scripted Tab walk, an axe scan per theme. Output is findings keyed to rule ids with reproducible evidence, plus the clearing re-run after a fix. - **IS NOT:** finding defects by reading source or deciding their tier and ship verdict (`ui-design` Audit mode owns both); building or restyling UI (`ui-design` Build); authoring a durable test suite (write Playwright tests); pixel-diff regression against a baseline (Chromatic, Percy); field performance (RUM or CrUX; Lighthouse is a lab tool). The division of labour is the point. A static audit reports what the code will probably do; it cannot see a 40px control whose hit area a pseudo-element already expands to 44, or a retry button wired to nothing. This skill reproduces or kills each of those, so a finding arrives with a measurement instead of a confidence. ## Contents - [When to run](#when-to-run) - [Probe catalogue](#probe-catalogue) - [Progress checklist](#progress-checklist) - [1. Establish the session](#1-establish-the-session) - [2. Select the probe set](#2-select-the-probe-set) - [3. Run the probes](#3-run-the-probes) - [4. Decide each finding](#4-decide-each-finding) - [5. Clearing re-run](#5-clearing-re-run) - [6. Report](#6-report) - [Honesty rules](#honesty-rules) - [Gotchas](#gotchas) - [Related skills](#related-skills) ## When to run | Situation | What this skill does | |---|---| | A `ui-design` audit emitted findings and the user wants them confirmed | Run only the probes those rule ids map to, in `references/rule-coverage.md` | | No audit ran; the user points at a route or a running app | Detect features, run the full battery on the resolved routes | | A fix just landed for a previously reproduced finding | Run the clearing re-run only (step 5) | | The user asks for captures across themes or widths | `probes/theme-locale-matrix.md` alone | | A `design-system.md` claims a scale and someone needs it checked | Read the claimed values off computed styles on a real page; a theme value the build overrides never reaches the browser | Nothing here assigns a tier or a ship verdict. Hand reproduced findings back with their evidence and let `ui-design`'s `references/ship-readiness.md` tier them; two skills tiering the same finding is how the tiers drift apart. ## Probe catalogue Each file is one probe: what it measures, the driver calls, the false positives it must guard, and the shape of the evidence it returns. | Probe | Measures | Primary for | |---|---|---| | [probes/axe-scan.md](./probes/axe-scan.md) | axe-core violations per route and theme, including computed contrast | contrast, accessible names, landmarks, document language | | [probes/target-size.md](./probes/target-size.md) | Bounding box and effective hit area of every visible interactive element | `interaction-target-size` | | [probes/focus-walk.md](./probes/focus-walk.md) | Scripted Tab traversal, focus-ring pixel delta, dialog trap and restoration | `focus-*`, `interaction-focus-visible`, `interaction-keyboard-operable` | | [probes/layout-shift.md](./probes/layout-shift.md) | Attributed `layout-shift` entries with the data response held open | `states-layout-shift`, `perf-image-dimensions-and-priority` | | [probes/viewport-stress.md](./probes/viewport-stress.md) | Horizontal overflow and clipped text at 320px and up, and under tripled strings | `layout-long-content-safety`, `mobile-viewport-scaling` | | [probes/failure-injection.md](./probes/failure-injection.md) | What renders when the data request returns 500, `[]`, malformed, or nothing | `states-no-error-state`, `states-no-empty-state`, `async-*`, `microcopy-*` | | [probes/theme-locale-matrix.md](./probes/theme-locale-matrix.md) | The same route captured across viewport, theme, pseudo-locale, and direction | `dark-i18n-untested`, `dark-i18n-rtl-untested` | | [probes/console-network.md](./probes/console-network.md) | Console errors, page errors, failed requests, hydration mismatch warnings | hydration mismatch, silent runtime failures | | [probes/web-vitals.md](./probes/web-vitals.md) | LCP with its attributed element, CLS, INP on a scripted interaction | perf attribution, never a budget verdict | ## Progress checklist ```text Verification progress: - [ ] Step 1: Establish the session (references/session-setup.md): driver, build mode, base URL, auth, resolved routes - [ ] Step 2: Select the probe set from handed-over rule ids (references/rule-coverage.md) or from the routes - [ ] Step 3: Run each probe; record evidence artifacts before interpreting any of them - [ ] Step 4: Decide each finding reproduced / not-reproduced / unknown; repeat timing-sensitive or inconsistent results when needed - [ ] Step 5: For each fix applied, re-run the identical probe and record clearedBy - [ ] Step 6: Emit the verification block per finding (references/evidence-output.md), then render - [ ] Step 7: List probes skipped and why. A probe that could not run is never a pass ``` ## 1. Establish the session Read [references/session-setup.md](./references/session-setup.md). It resolves four things, and getting any of them wrong invalidates every probe downstream: - **Driver.** Playwright when the repo can run it, the Chrome DevTools browser tools when it cannot. Four probes need request interception and network control, so they do not exist under a driver without them. - **Build mode.** Perf and layout-shift probes run against a production build; dev-mode numbers measure the bundler. Failure injection and focus probes run fine in dev. - **Auth and seeded data.** A route behind a login with no seeded session returns `unknown`, never a pass. - **Routes.** Map the diff to URLs. A component with no reachable route falls back to the repo's Storybook, and to `unknown` if there is none. Stop here if the app will not boot. A verification run with no session produces no findings, which is a reportable outcome and not a clean bill of health. ## 2. Select the probe set **Handed a list of rule ids** (the normal case, from a `ui-design` audit): open `references/rule-coverage.md`, take the probe each id maps to, and run only those. Ids with no probe stay source-only findings and pass through untouched, marked as such. **Given only routes:** detect features the way `ui-design`'s feature playbooks do (form, list, modal, dashboard, checkout), then run the battery that surface earns. `probes/axe-scan.md`, `probes/console-network.md`, and `probes/viewport-stress.md` run on every route regardless: they are cheap, and they are the three that find things nobody suspected. Budget the matrix before running it. Routes multiplied by viewports multiplied by themes grows fast, and a run that takes twenty minutes gets skipped next time. Two viewports (360 and 1280) and two themes cover the ground; add widths only where a probe already found an edge. ## 3. Run the probes Each probe file carries its own recipe. Three rules hold across all of them: - **Capture evidence before interpreting it.** Write the screenshot, the JSON measurement, and the console log to disk first. A finding whose evidence was never written is unverifiable by the person reading the report, which puts it back where the static audit left it. - **One probe, one route, one viewport, one theme.** Never fold two conditions into one run: when the result surprises you, you need to know which axis produced it. - **Repeat uncertain measurements.** Retry timing-sensitive or inconsistent results under controlled conditions. A deterministic failure with a captured trigger needs no duplicate run. A result that flips remains inconclusive until its conditions are understood. ## 4. Decide each finding Three outcomes, and the middle one is the one that earns this skill its keep. | Outcome | Meaning | What it does to the handed-over finding | |---|---|---| | `reproduced` | The probe measured the defect | Stays a `fail`, now carrying `observed` from the measurement rather than from the source read | | `not-reproduced` | The tested conditions did not exhibit the defect | Withdraw only if the probe exercised the alleged trigger; otherwise retain the candidate with the remaining coverage gap | | `unknown` | The probe could not run or could not decide | Finding survives as `unknown` with the probe's reason. Never converts to a pass | A `not-reproduced` result is a real deliverable, not a wasted run. It is what removes the false positives a large rule corpus asserts with `file:line` confidence, and it feeds the rejection section `ui-design` already requires. Where a probe finds something no static rule predicted, emit it as a new finding against the rule id the probe is primary for. Where no rule covers it (an axe violation with no ui-design counterpart, a console error), emit it under `axe:<violation-id>` or `runtime:<signature>` and say plainly that it came from the browser and not the corpus. ## 5. Clearing re-run A fix is not verified by reading the diff. Re-run the identical probe: same route, same viewport, same theme, same seed, same injected failure. Record it as `clearedBy` on the finding, keep the before-artifact, and never overwrite it with the after. Three outcomes worth naming: - The probe now passes: the finding is `applied` and cleared. - The probe still fails: the fix did not work. Report it as applied-unverified and say so; do not report the fix and stay silent about the re-run. - The probe passes but a different probe on the same route now fails: the fix caused a regression, which is a new finding, not a footnote on the old one. ## 6. Report [references/evidence-output.md](./references/evidence-output.md) owns the shape: a `verification` block appended to each finding, a top-level `session` block, and artifact paths. It defines only the delta on `ui-design`'s `references/output-adapters.md` schema, which stays the single owner of the finding object, the three counts, and the verdict. Where this skill runs standalone, render the same terminal adapter with `SHIP VERDICT` omitted: the verdict is a property of a tiered audit, and printing one from probe results alone invents a tier assignment nobody made. ## Honesty rules - **A skipped probe is not a passed probe.** Report every probe that did not run, with the reason (no route, auth required, driver lacks interception, app would not build). A report that silently omits them reads as coverage it does not have. - **A screenshot is evidence, not a verdict.** Probes decide on measurements. Captures exist so a human can check the measurement, and for the two questions no measurement settles: whether the dark theme looks right, and whether the pseudo-locale broke the layout or merely the prose. - **Do not widen into a redesign.** This skill reports what it measured. A finding that needs a new type scale names the mode to run next, exactly as an audit does. - **The app is the subject, not the harness.** A failure caused by the probe itself (a selector that never resolved, a route interception that swallowed the wrong request) is a harness bug. Fix the probe and re-run; never report it as a defect in the app. - **Numbers carry their units and their conditions.** `44x44px at 360px width with touch emulation on` is a measurement. `too small` is the inference this skill exists to replace. ## Gotchas The ones that cut across probes. Each probe file carries its own false positives. - `page.emulateMedia({ colorScheme: 'dark' })` does nothing for an app that themes with a `class="dark"` or `data-theme` attribute, which is most Tailwind apps. The probe reports a clean dark pass while never having left light mode. Drive the app's own toggle, and assert the attribute landed before capturing. - Running perf or layout-shift probes against `next dev` measures on-demand compilation. The first navigation to a route can spend seconds in the bundler, which lands in LCP and dwarfs anything real. - Animations that never settle keep a screenshot probe waiting until timeout. Set `prefers-reduced-motion: reduce` for captures, then run motion-sensitive checks in a separate pass with it off, because reduced motion is also a code path that can be broken. ## Related skills - `ui-design`: reads source, produces the findings this skill reproduces, and owns tiering, the ship verdict, and the finding schema. - `typography-audit`: type findings that need a rendered measure or leading value can be handed here for the measurement. - `ax-audit`: agentic surfaces. Its runtime questions use the same session and probes. - `ui-animation`: motion craft. This skill can capture the timing, but judging the curve is that skill's. Maintenance only: `evals/evals.json` holds the behavioural scenarios and routing prompts for anyone changing this skill. It never loads during a verification run.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.