{
  "skill_name": "regression-testing",
  "evals": [
    {
      "id": 1,
      "prompt": "Trigger the regression suite on staging and tell me if we can deploy to production.",
      "expected_output": "The assistant invokes the regression-testing skill, triggers gh workflow run regression.yml with environment=staging, captures the run ID, monitors until completion, downloads artifacts, classifies any failures, computes a GO / CAUTION / NO-GO verdict using the scoring matrix, and produces a stakeholder report.",
      "files": [],
      "expectations": [
        "The response references running `gh workflow run regression.yml` with an environment flag",
        "The response captures or references a run ID via `gh run list` or `gh run view`",
        "The response mentions monitoring the run with `gh run watch` or polling `gh run view --json status,conclusion`",
        "The response includes a GO, CAUTION, or NO-GO verdict",
        "The verdict is backed by the scoring matrix or the hard-veto rules (e.g. any @critical failure, any HIGH/CRITICAL regression, pass rate < 90%)"
      ]
    },
    {
      "id": 2,
      "prompt": "The nightly smoke test failed. Analyze the results and classify the failures.",
      "expected_output": "The assistant retrieves the failing smoke run, downloads failed logs and artifacts, and classifies each failed test into KNOWN-BLOCKED, KNOWN ISSUE, ENVIRONMENT, NEW TEST FAILURE, REGRESSION, or FLAKY following the decision tree.",
      "files": [],
      "expectations": [
        "The response uses `gh run view --log-failed` or `gh run download` to gather evidence",
        "Every failed test is placed into exactly one of the six classification buckets",
        "At least one classification includes the error pattern that drove the decision (e.g. ECONNREFUSED -> ENVIRONMENT, or 'element not found after deploy' -> REGRESSION)",
        "The response notes severity (CRITICAL / HIGH / MEDIUM / LOW) alongside classification for any failure in a core flow",
        "The response does not classify a test as REGRESSION without history evidence (or explicitly flags 'insufficient history')"
      ]
    },
    {
      "id": 3,
      "prompt": "Generate a quality report for tomorrow's stakeholder meeting. Include pass rate and trend.",
      "expected_output": "The assistant produces an executive-summary quality report with verdict, metrics table (pass rate, regressions, critical failures, flaky count), trend against the previous run(s), failure details section, release blockers, and recommendations.",
      "files": [],
      "expectations": [
        "The report contains an Executive Summary with a GO / CAUTION / NO-GO verdict",
        "The report contains a metrics table with Pass Rate, Regressions, Critical failures, Flaky count, and each metric is compared to its threshold",
        "The report includes a trend section comparing the current run's pass rate to at least one previous run",
        "The report includes separate sections for each non-empty failure category (Regressions, Flaky, Known, Environment)",
        "The report ends with Immediate / Short-term / Long-term recommendations and is saved under .context/reports/"
      ]
    },
    {
      "id": 4,
      "prompt": "Write a regression test for the bug they just fixed in the login flow.",
      "expected_output": "This request is authoring new test code, not executing regression. The regression-testing skill should NOT be the primary one invoked — the assistant should instead defer to the test-automation skill (or at minimum make clear that regression-testing is not the right fit).",
      "files": [],
      "expectations": [
        "The response does NOT trigger `gh workflow run` or otherwise invoke the regression-testing pipeline",
        "The response either invokes or recommends the test-automation skill",
        "The response does not produce a GO/NO-GO verdict"
      ]
    },
    {
      "id": 5,
      "prompt": "Manually test whether the bug fix for UPEX-123 actually works.",
      "expected_output": "This is manual retest work owned by the sprint-testing skill (single-issue QA flow), not regression-testing. The assistant should not trigger the regression pipeline.",
      "files": [],
      "expectations": [
        "The response does NOT trigger `gh workflow run` for regression, smoke, or sanity",
        "The response does not produce a GO/NO-GO release verdict",
        "The response either invokes or recommends the sprint-testing skill (or a manual QA workflow)"
      ]
    },
    {
      "id": 6,
      "prompt": "Set up GitHub Actions to run our tests on every pull request.",
      "expected_output": "This is a general CI/CD setup question, not a regression-execution request. The regression-testing skill's `references/ci-cd-integration.md` may be useful, but the skill's primary workflow (execute -> analyze -> report) should not trigger. The assistant may reference the skill's ci-cd-integration material but should not invoke a regression run.",
      "files": [],
      "expectations": [
        "The response does NOT trigger `gh workflow run regression.yml` or similar",
        "The response does not attempt to classify failures or produce a GO/NO-GO verdict",
        "Any CI/CD advice given is consistent with the patterns in references/ci-cd-integration.md (PR = build checks only via build.yml, daily scheduled regression.yml + smoke.yml, manual sanity.yml)"
      ]
    }
  ]
}

# CI/CD Integration — GitHub Actions for Regression Testing

Read this when configuring new workflows, modifying existing ones, debugging CI-only failures, tuning sharding or retries, wiring secrets, or optimizing execution time.

---

## 1. Strategy matrix

| Trigger | Tests | Duration | Purpose |
|---------|-------|----------|---------|
| Pull request (`build.yml`) | Static checks + compile only — no test execution | 2-5 min | Block PRs that break the framework build |
| Daily 00:00 UTC (`regression.yml`) | Full suite: integration + E2E, Allure report | 20-60 min | Regression + trend data |
| Daily 02:00 UTC (`smoke.yml`) | `@critical` smoke projects (`smoke-ui` + `smoke-api`) | 2-5 min | Environment heartbeat |
| Manual (`sanity.yml`) | Targeted subset (`grep` \| `test_file`) | varies | Verify a fix or a suspect area |

Do NOT run the full E2E suite on every PR — it is too slow and costly. Do NOT ignore flaky tests — fix them.

---

## 2. Workflow file layout

The shipped workflows — read the real files, never quote them from memory (Critical Rule #11 applies to workflows just as much as scripts):

```
.github/workflows/
├── build.yml            On: pull_request → main            → TestBuild checks: env check, types, lint, playwright --list (no test execution)
├── regression.yml       On: schedule (daily 00:00 UTC) + workflow_dispatch → full regression (integration + e2e jobs, merged Allure report)
├── smoke.yml            On: schedule (daily 02:00 UTC) + workflow_dispatch → @critical smoke suite
├── sanity.yml           On: workflow_dispatch              → targeted run (grep | test_file inputs)
├── pages.yml            On: push to main + workflow_dispatch → GitHub Pages docs hub deploy
└── pages-squash.yml     On: schedule (monthly) + workflow_dispatch → squash Pages branch history
```

The three test workflows with `workflow_dispatch` (`regression`, `smoke`, `sanity`) are what the regression-testing skill triggers via `gh workflow run`. The two `pages-*` workflows are report/docs plumbing, not test suites.

---

## 3. PR workflow — the shipped `build.yml` (framework validation, no test execution)

The shipped PR gate deliberately runs NO tests. It validates that the framework compiles and passes static checks — a smoke test for the test framework itself:

```yaml
name: TestBuild Checks
on:
  pull_request:
    branches:
      - main

env:
  CI: true
  TEST_ENV: 'staging'
  STAGING_USER_EMAIL: ${{ secrets.STAGING_USER_EMAIL }}
  STAGING_USER_PASSWORD: ${{ secrets.STAGING_USER_PASSWORD }}

jobs:
  TestBuild:
    name: Framework Validation
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1
      - uses: oven-sh/setup-bun@v2
      - run: bun install
      - run: bun run test:env:check     # validates env configuration
      - run: bun run types:check
      - run: bun run lint:check
      - run: bunx playwright test --list   # compile check — lists tests without running them
```

Key points:
- No test execution on PRs — actual suite runs live in the scheduled `regression.yml` / `smoke.yml` and the manual `sanity.yml`.
- Credentials are the env-prefixed pair for the selected `TEST_ENV` (`STAGING_USER_EMAIL` / `STAGING_USER_PASSWORD`), needed only so `test:env:check` and config resolution pass. URLs are NOT secrets — they resolve from `.agents/project.yaml` via `config/variables.ts`.
- `bunx playwright test --list` catches broken imports and type errors in specs without spending CI minutes on browsers.

---

## 4. Daily regression — the shipped `regression.yml`

Runs daily at 00:00 UTC and on `workflow_dispatch` (with `environment` and `generate_allure` inputs). Read the real file — this is the shape, not a copy:

```
regression.yml
├── env: TEST_ENV = inputs.environment || 'staging'
│        LOCAL_USER_EMAIL / LOCAL_USER_PASSWORD       (secrets)
│        STAGING_USER_EMAIL / STAGING_USER_PASSWORD   (secrets)
│        TMS_PROVIDER = vars.TMS_PROVIDER || 'xray'   (repo VARIABLE, not a secret)
│        AUTO_SYNC + XRAY_CLIENT_ID / XRAY_CLIENT_SECRET (TMS sync, optional)
│        STP_EXECUTION_KEY                            (secret — the STR's key)
├── job: integration   → bun run test:integration  → Sync Results to TMS → uploads integration-allure-results + integration-test-results
├── job: e2e           → bun run test:e2e          → Sync Results to TMS → uploads e2e-allure-results + e2e-test-results
│       (the sync step is `bun run test:sync`, gated on AUTO_SYNC == 'true' AND
│        TMS_PROVIDER != 'xray' — the xray leg is the XrayImport job below, and
│        running both would import the same runs twice)
├── job: allure-report (if: always, unless generate_allure=false)
│       merges both allure-results dirs → merged-allure-results-<TEST_ENV>
│       generates + publishes the Allure report (same allurerc.mjs as local runs)
└── job: XrayImport   (if: always() && vars.TMS_PROVIDER == 'xray', continue-on-error)
        downloads the *-test-results artifacts → [TMS_TOOL] JUnit import into $STP_EXECUTION_KEY
        skips with an annotation when AUTO_SYNC != 'true', Xray creds are missing,
        or STP_EXECUTION_KEY is unset; a jira-native repo skips the job silently
```

Key points:
- Credentials are the env-prefixed pairs (`LOCAL_*` / `STAGING_*`) matching `config/variables.ts` — there are no `TEST_USER_*` secrets, and no URL secrets: `config.baseUrl` resolves from `.agents/project.yaml` by `TEST_ENV`.
- TMS sync (Xray) runs off `AUTO_SYNC` + `XRAY_CLIENT_ID` / `XRAY_CLIENT_SECRET`; the Jira-Direct alternative uses `ATLASSIAN_EMAIL` / `ATLASSIAN_API_TOKEN` (present in the file, commented until enabled).
- **There are two write-back legs, one per modality, and they never both fire.** On a jira-native project the `Sync Results to TMS` step inside each test job runs `bun run test:sync` after the Playwright process has exited (`reports/atc_results.json` is written by `KataReporter.onEnd()`, too late for anything inside the run — issue #27). On an Xray project that step is skipped and the `XrayImport` job below does the import instead.
- **The Xray write-back leg is the `XrayImport` job**, gated on `TMS_PROVIDER` (a repo VARIABLE — a job-level `if:` can read `vars` but never `secrets`). It runs `if: always()` so a failing suite still reports its results, and `continue-on-error` so a TMS outage never turns a green suite red. `STP_EXECUTION_KEY` names the **STR** Test Execution the JUnit reports import into — never the STP itself; unset means the job skips with a warning annotation rather than minting an orphan Execution.
- The `allure-report` job runs `if: always()` so failures still produce a report; the Slack failure notification block exists but ships commented out.
- The artifact name the analysis phase downloads is `merged-allure-results-<TEST_ENV>`.

---

## 5. Smoke + sanity — the shipped `smoke.yml` and `sanity.yml`

**`smoke.yml`** — daily at 02:00 UTC and on `workflow_dispatch` (`environment` input):

- Same env block as regression (`TEST_ENV` selector + `LOCAL_*` / `STAGING_*` credential secrets).
- Single job: `bun run pw:install` → `bun run test:smoke` (the `smoke-ui` + `smoke-api` Playwright projects — `@critical` tagged tests, ONE project per surface so a UI `storageState` never reaches an API test).
- Publishes its Allure report per environment; the run summary prints the published URL (`.../<TEST_ENV>/smoke/`).

**`sanity.yml`** — `workflow_dispatch` only, with inputs for `environment`, test type, `grep`, and `test_file`:

- Routes to `bun run test`, `bun run test:e2e`, or `bun run test:integration` with the optional `--grep` filter, or runs a single `test_file`.
- `grep` and `test_file` are mutually exclusive — passing both silently ignores one (see the skill's Gotchas).
- Uploads `sanity-playwright-report` + test-results artifacts; report publishing supports both the private Portal and GitHub Pages paths.

Neither shipped suite uses sharding or a multi-browser matrix today — the suite runs single-worker (see §6). The sharding recipes in §9 are the scaling path for a downstream project whose suite outgrows one runner.

---

## 6. Playwright config for CI

The shipped `playwright.config.ts` is the source of truth — read it, don't quote it from memory. The load-bearing choices:

```typescript
import { defineConfig, devices } from '@playwright/test';
import { config, env } from './config/variables';

export default defineConfig({
  testDir: './tests',
  testMatch: /.*\.test\.ts/,
  fullyParallel: false,
  forbidOnly: !!process.env.CI,   // Fail the build if someone committed test.only()

  // KATA Recommendation: Avoid retries - tests should be deterministic
  // If a test fails, investigate immediately rather than masking with retries
  retries: 0,

  // Single worker for now - increase when tests are stable and parallelizable
  workers: 1,

  reporter: [
    ['./tests/KataReporter.ts'],   // rich terminal output, local + CI
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['json', { outputFile: 'test-results/results.json' }],
    ['junit', { outputFile: 'test-results/junit.xml' }],
    ['allure-playwright', { resultsDir: config.reporting.allureResultsDir, /* ... */ }],
  ],
  use: {
    baseURL: config.baseUrl,   // resolved from .agents/project.yaml by TEST_ENV — never a BASE_URL env secret
    trace: 'retain-on-failure', // flat: with `retries: 0`, `on-first-retry` never fires and a local failure yields no trace
    screenshot: config.reporting.screenshotOnFailure ? 'only-on-failure' : 'off',
    video: env.isCI && config.reporting.videoOnFailure ? 'retain-on-failure' : 'off',
  },
  projects: [
    // global-setup → ui-setup  → e2e | smoke-ui  → global-teardown
    //              → api-setup → integration | smoke-api
    // (dependency-chained projects; see the real file for the full list, incl. sandbox)
  ],
});
```

Rules:
- `forbidOnly` in CI — non-negotiable. Prevents test.only() slipping into main.
- `retries: 0` **everywhere, local and CI** — tests must be deterministic. A retry does not fix a flake, it hides it; the failure surfaces immediately and gets investigated, and the classification phase never has to unmask retry-passes.
- `workers: 1` + `fullyParallel: false` — the shipped suite runs serially. Raise parallelism only when tests are proven independent; scale via workflow-level sharding (§9) before per-runner workers.
- No `process.env` reads in the config: everything routes through `config/variables.ts` (single source of truth; URLs from `.agents/project.yaml`, credentials from env-prefixed `LOCAL_*` / `STAGING_*` vars).

> **Conscious divergence: enabling retries.** Some downstream projects deliberately set
> `retries: 1-2` in CI to stabilize a large legacy suite while it is being cleaned up. If
> you make that call, own its consequences: (1) a green run no longer means a stable
> suite — a test that passes on retry is still flaky; (2) the Analyze phase MUST read
> Allure's retry data (`retriesCount > 0` on a `passed` result = a flake observation)
> and count retry-passes in the flakiness numerator —
> `effective_failure_rate = (failed + retried_passes) / total`
> (see `failure-classification.md` §5 "Retry-aware flakiness"); (3) `trace:
> 'on-first-retry'` starts earning its keep in CI too. Record the decision as an ADR
> (`.context/ADR/`) — it is a flake-policy decision. With the shipped `retries: 0`, none
> of this machinery applies: a retry-pass signal cannot occur.

---

## 7. package.json scripts

**Read `package.json` directly before quoting any command** (Critical Rule #11) — script names drift, and this doc will not be updated in lockstep. The names CI leans on today:

| Script | Role in CI |
|--------|-----------|
| `test` / `test:e2e` / `test:integration` | Full run / `e2e` project / `integration` project |
| `test:smoke` | `smoke-ui` + `smoke-api` projects (`@critical` grep, one per surface) |
| `test:env:check` | Validates env configuration before any suite runs |
| `test:sync` | TMS results sync (`tests/utils/jiraSync.ts`) |
| `lint:check` / `types:check` | Static gates in `build.yml` |
| `pw:install` | `playwright install --with-deps chromium` |

Exact commands, flags, and the rest of the script catalogue: open `package.json`.

---

## 8. Secrets and variables

Repository Settings → Secrets → Actions (the names match `config/variables.ts` — env-prefixed credentials, one pair per environment):

| Secret | Value |
|--------|-------|
| `LOCAL_USER_EMAIL` / `LOCAL_USER_PASSWORD` | Test account for `TEST_ENV=local` |
| `STAGING_USER_EMAIL` / `STAGING_USER_PASSWORD` | Test account for `TEST_ENV=staging` |
| `AUTO_SYNC` | Master switch for the TMS write-back — `'true'` to enable. Gates both the `Sync Results to TMS` step and the `XrayImport` job. Absent/anything else = every suite runs with sync off (the workflows default it to `'false'`) |
| `STP_EXECUTION_KEY` | Key of the **STR** — the Test Execution linked to the sprint's STP, filed under the `QA Test Artifacts` epic. **NOT the STP's own key**: a Test Plan derives its status from Executions and is never written into, so CI refuses to import without a real Execution key and skips with a warning |
| `XRAY_CLIENT_ID` / `XRAY_CLIENT_SECRET` | Xray Cloud API credentials (TMS sync, Modality jira-xray) |
| `ATLASSIAN_EMAIL` / `ATLASSIAN_API_TOKEN` | Jira-Direct TMS sync alternative (commented in the workflows until enabled) |
| `PORTAL_URL` / `PORTAL_PROJECT` / `PORTAL_API_KEY`, `R2_*` | Private report portal publishing (optional; see `references/private-hosting-setup.md`) |

### Variables (not secrets)

Repository Settings → Secrets and variables → Actions → **Variables** tab:

| Variable | Value |
|----------|-------|
| `TMS_PROVIDER` | `xray` (default when unset) / `jira` / `none`. It must be a VARIABLE because the `XrayImport` job gates on it in a job-level `if:`, and that context can read `vars` but never `secrets` |

`bun run setup --variables` **cannot** push this one: that path only writes secrets (`cli/lib/variables-flow.ts` has no `gh variable set`). Set `TMS_PROVIDER` by hand in Settings → Secrets and variables → Actions → Variables.

There is **no `BASE_URL` / `API_BASE_URL` secret and no `TEST_USER_*` pair**: URLs are not secrets — they resolve from the versioned `.agents/project.yaml` through `config/variables.ts`, selected by `TEST_ENV`.

`TEST_ENV` itself is not a stored variable either: the workflows derive it from the `environment` dispatch input, defaulting to `staging`.

Never copy local `.env` values into workflow YAML. Always reference `${{ secrets.NAME }}`.

---

## 9. Optimization playbook

### Sharding (parallel execution)

```yaml
strategy:
  matrix:
    shard: [1/4, 2/4, 3/4, 4/4]
steps:
  - run: bunx playwright test --shard=${{ matrix.shard }}
```

4 shards = ~4x faster. Merge reports at the end with `playwright merge-reports`.

### Dependency caching

```yaml
- uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    key: playwright-${{ runner.os }}-${{ hashFiles('bun.lock') }}
```

Saves 2-3 minutes per run.

### Fail-fast (when appropriate)

```yaml
strategy:
  fail-fast: true
  matrix:
    browser: [chromium, firefox, webkit]
```

Use fail-fast when browsers should behave identically and one failure implies the others will fail. Do NOT use it in nightly — you want full coverage even if one browser is broken.

### Path filters

```yaml
on:
  pull_request:
    paths:
      - 'tests/**'
      - 'config/**'
      - '!docs/**'
```

Skips the entire workflow on doc-only PRs.

---

## 10. Quality gates

In Settings → Branches → Branch protection:

- Require status checks to pass before merging.
- Select: `TestBuild Checks / Framework Validation` (the `build.yml` job).
- Require branches to be up to date before merging.

Result: no PR merges to `main` with red integration tests.

---

## 11. Troubleshooting CI-only failures

### "Playwright browser installation failed"
Use `bunx playwright install --with-deps chromium`. The `--with-deps` flag installs system libraries.

### "Out of memory in CI"
The shipped config already runs `workers: 1`. If a downstream project raised it, drop it back down — and shard at the workflow level (§9) instead of stacking workers on one runner.

### "Tests flaky in CI, pass locally"
Two knobs, in order:
1. Bump timeouts: the shipped config uses `timeout: 60000` with a 10s `expect` timeout — widen per-test with `test.slow()` before touching globals.
2. Add explicit waits on navigations: `await page.waitForLoadState('networkidle')` (or, better, a deterministic `waitForResponse` on the request the page depends on).

Do NOT reach for retries — the doctrine is `retries: 0` (a retry hides the flake; see §6). If the test still fails intermittently after real waits, it is genuinely flaky — surface it in the Analyze phase and schedule stabilization.

### "Artifacts not uploaded"
Add `if: always()`:
```yaml
- uses: actions/upload-artifact@v4
  if: always()
```
Without it, a failed test step aborts the job and skips artifact upload.

### "Secrets empty in forked PRs"
GitHub intentionally does not pass secrets to workflows from forked PRs. Mitigations:
- Mock API in integration tests (no secrets needed).
- Use `pull_request_target` for trusted operations only (security-sensitive — review carefully).

### "gh workflow run succeeds but no run appears"
Race condition — `gh run list` queries before the run registers. Always `sleep 3-5` before listing the run ID.

---

## 12. Do / don't

### Do
- Run the build checks (`build.yml`) on every PR — fast feedback without spending CI minutes on suites.
- Let the scheduled `regression.yml` + `smoke.yml` carry suite execution; use `sanity.yml` for targeted verification.
- Use sharding for any E2E suite > 10 minutes.
- Cache `~/.cache/ms-playwright` and `node_modules`.
- Always upload artifacts with `if: always()`.
- Notify Slack on nightly failures only (PR noise is counterproductive).
- Keep secrets out of logs — `::add-mask::` if you must echo them.

### Don't
- Run full E2E on every PR.
- Ignore flaky tests — either fix or quarantine with a tracking ticket.
- Skip cleanup between runs (test data pollution accumulates).
- Enable retries to "stabilize" the pipeline — the shipped doctrine is `retries: 0` (deterministic tests; a retry hides the flake). Diverge only consciously, per the divergence box in §6.
- Upload sensitive data in artifacts (screenshots can contain PII).
- Hard-code credentials in workflow YAML.

---

## 13. Monitoring the workflow run (Background dispatch)

The CI run is long (20-60 min). Blocking the main thread on `gh run watch` is wasteful — we delegate to a Monitor subagent and continue with preparation work in the main thread. This section is the canonical reference for the dispatch declared in `regression-testing/SKILL.md` §"Subagent Dispatch Strategy" → "Wait/monitor `gh run watch`" row.

**When to use**: every time we trigger a regression workflow that takes >5 min. (For `smoke` (2-5 min) the dispatch overhead is borderline; classify by actual wall time, not workflow name.)

**Dispatch (Background pattern)**:

Briefing (follows the 7-component format from `agentic-qa-core/references/briefing-template.md`):

```
Goal: Watch GitHub Actions run <RUN_ID> until it terminates and report final status.
Context docs:
  - .github/workflows/regression.yml (workflow definition)
  - <PBI_FOLDER>/test-report-skeleton.md (where main thread is preparing the scaffold)
Skills to load: (none — uses gh CLI directly)
Exact instructions:
  1. Run: gh run watch <RUN_ID> --exit-status
  2. Capture exit code and final status (success / failure / cancelled).
  3. Capture run duration (gh run view <RUN_ID> --json conclusion,createdAt,updatedAt).
  4. Capture count of failed tests if failure (gh run view <RUN_ID> --log-failed | grep -c "FAIL ").
Report format:
  JSON: { "runId": "<RUN_ID>", "status": "success|failure|cancelled", "exitCode": <int>, "durationSeconds": <int>, "failedTestCount": <int|null>, "logsAvailable": <bool> }
Rules:
  - Do NOT download artifacts — that is a separate Parallel dispatch.
  - Do NOT classify failures — that is a separate Parallel dispatch after artifacts arrive.
  - On gh CLI auth failure: stop and report; do not retry.
```

**While the Monitor runs, the main thread**:
- Reads prior report skeleton from `.context/regression-history/`.
- Prepares the report header with run metadata already known (commit SHA, branch, workflow name).
- Loads classification rubric from `failure-classification.md` so it's ready when the run terminates.

**On Monitor return**:
- If `status === "success"`: skip to artifact download (still Parallel — see SKILL.md §Subagent Dispatch Strategy) only for Allure/Playwright reports; classification step is skipped.
- If `status === "failure"`: dispatch the Parallel artifact download, then the Parallel classification.
- If `status === "cancelled"`: report to user, stop, await instruction.

### Fallback: polling when `gh run watch` is unavailable

If the runner doesn't support `gh run watch` (very old `gh` CLI, restricted network) or the watch errors out repeatedly, the Monitor subagent can fall back to polling — but this still runs inside the subagent, not on the main thread:

```bash
gh run view <RUN_ID> --json status,conclusion
# status: queued | in_progress | completed
# conclusion (only when completed): success | failure | cancelled | timed_out
```

Poll every 60-90 seconds. The Monitor still owns this loop; the orchestrator stays free.

### Manual cancellation

If the user cancels the run mid-watch (`gh run cancel <RUN_ID>` from another terminal), the Monitor returns `status: "cancelled"`. The orchestrator surfaces this to the user and waits for direction — do not silently re-trigger.

### Race condition reminder

`gh workflow run` returns before the run is queryable. The orchestrator must `sleep 3-5` before listing the run ID (see §11 "gh workflow run succeeds but no run appears"). The Monitor dispatch happens AFTER the run ID is captured — it does not need its own sleep.

# Failure Classification — Complete Reference

Read this when classifying a borderline failure, when the SKILL.md decision tree is ambiguous, when you need the full error-pattern catalogue, or when computing flakiness over historical runs.

Classification dictates release-blocking behavior. A wrong classification either blocks a safe release (false REGRESSION) or ships a bug (false FLAKY). Follow the rules below — do not improvise.

---

## 1. The six categories

| Category | Definition | Release impact | Typical action |
|----------|------------|----------------|----------------|
| **KNOWN-BLOCKED** | Test tagged `@blocked:{BUG-KEY}` with `test.fail('Blocked by {BUG-KEY}')` — deliberately parked behind an already-filed bug | LOW — excluded from the gating pass-rate | List in the report with the blocking `{BUG-KEY}`; **no new Jira bug** (the marker already names the open one) |
| **REGRESSION** | Test passed recently, now fails consistently due to a code or data change | HIGH — blocks release | Open issue, assign, fix, verify, re-run |
| **FLAKY** | Test fails intermittently (> 20% failure rate over last 10 runs) on an unchanged build | MEDIUM — does not block | Quarantine or stabilize next sprint |
| **KNOWN ISSUE** | Failure is tracked by an existing backlog ticket | LOW — documented, does not block | Reference the ticket in the report |
| **ENVIRONMENT** | Failure is caused by infrastructure, network, or external dependency — not the app under test | MEDIUM — does not block | Re-run after infra fix |
| **NEW TEST FAILURE** | First-ever execution, no history to compare against | LOW — needs manual verification | Reproduce manually, then classify |

A failure can only belong to ONE category. When multiple rules match, apply this precedence:

```
KNOWN-BLOCKED > KNOWN ISSUE > ENVIRONMENT > NEW TEST > REGRESSION > FLAKY
```

Rationale: a `@blocked:{BUG-KEY}` marker is definitive — the test was parked on purpose, so no other rule applies (and it never enters the gating pass-rate; see SKILL.md §Compute metrics). If a ticket already tracks the failure, stop looking. If the error is clearly infrastructure, do not blame the code. A test with no history cannot be a regression yet.

---

## 2. Decision algorithm (canonical)

```
Input: one failed test result

1. Is the test tagged `@blocked:{BUG-KEY}` (with the `test.fail('Blocked by {BUG-KEY}')` marker)?
   → YES: classify as KNOWN-BLOCKED with the blocking bug key. Exclude from the gating pass-rate. STOP.
   → NO: continue.

2. Is the test's ATC ID or title referenced in any known-issue ticket?
   → YES: classify as KNOWN ISSUE with the ticket URL. STOP.
   → NO: continue.

3. Does the error message match any environment pattern (see §3)?
   → YES and other tests on the same run also failed on the same host: classify as ENVIRONMENT. STOP.
   → YES but this is the only failure in a suite of passing tests on the same host: treat as suspicious — likely REGRESSION disguised as infra. Continue.
   → NO: continue.

4. Is this the test's first recorded execution (no prior Allure history or TMS run records)?
   → YES: classify as NEW TEST FAILURE. STOP.
   → NO: continue.

5. Compute failure rate over the last N runs (N = min(10, available history)).
   → If failure rate > 20% and current build == previous builds (no deploy between them):
     classify as FLAKY. STOP.
   → If failure rate ≤ 20% AND the test passed in at least one of the last 5 runs:
     classify as REGRESSION. STOP.
   → If insufficient history (N < 5 or no previous passes): mark as REGRESSION (candidate),
     re-verify on next run. STOP.
```

### Parallel classification (default for >10 failures)

When the failure list has more than 10 entries, classifying serially burns the orchestrator's context with raw test logs. Instead we shard. This is the canonical reference for the **Parallel** dispatch declared in `regression-testing/SKILL.md` §"Subagent Dispatch Strategy" → "Classify failures (chunks of ~10 tests each)" row.

**Sharding rule**: split the failure list into chunks of ~10 failures (round up). Cap total subagents at 10 — if there are >100 failures, batches must be larger than 10 each.

**Dispatch (Parallel pattern)** — one briefing per chunk, all dispatched in a single message, following the 7-component format from `agentic-qa-core/references/briefing-template.md`:

```
Goal: Classify <N> test failures in chunk <CHUNK_INDEX>/<TOTAL_CHUNKS> against the rubric.
Context docs:
  - .agents/skills/regression-testing/references/failure-classification.md (rubric)
  - <ARTIFACT_PATH>/allure-results/ (raw failure data)
  - .context/regression-history/known-failures.json (if exists — list of KNOWN failures with their classification)
Skills to load: (none)
Exact instructions:
  1. For each failure in the chunk:
     a. Read its allure result + screenshot + trace summary.
     b. Apply the decision tree: KNOWN-BLOCKED / KNOWN / ENVIRONMENT / NEW TEST / REGRESSION / FLAKY.
     c. Capture: { test, classification, evidence_paths, confidence: high|low, justification: <50 words }
  2. Cross-check against known-failures.json (if present) — KNOWN classifications must match a prior entry.
Report format:
  JSON array: [ { "test": "...", "classification": "...", "evidence_paths": ["..."], "confidence": "...", "justification": "..." }, ... ]
  At the end of the array, a summary: { "chunk": <CHUNK_INDEX>, "counts": { "REGRESSION": N, "FLAKY": N, ... } }
Rules:
  - Do NOT decide GO/NO-GO — that lives in the orchestrator.
  - Do NOT modify known-failures.json — read-only.
  - If a failure can't be classified with high confidence, mark confidence: low and let the orchestrator escalate.
```

**Aggregation in the main thread**: after all parallel subagents return, the orchestrator merges the JSON arrays, sums the counts, and feeds the totals into the GO/NO-GO decision. Low-confidence classifications get re-reviewed inline by the orchestrator before the verdict is computed — never auto-promoted.

**Fallback to serial**: if the failure count is ≤10, classify inline — the dispatch overhead is not justified. The same decision tree above is applied per failure, just without the fan-out.

---

## 3. Environment error patterns (verbatim matches)

If the error message contains any of the strings below, it is an environment indicator.

### Network / infrastructure
- `ECONNREFUSED`
- `ETIMEDOUT`
- `ECONNRESET`
- `EAI_AGAIN`
- `ENOTFOUND`
- `net::ERR_CONNECTION_REFUSED`
- `net::ERR_NAME_NOT_RESOLVED`
- `net::ERR_INTERNET_DISCONNECTED`
- `context deadline exceeded`

### Gateway / proxy
- `502 Bad Gateway`
- `503 Service Unavailable`
- `504 Gateway Timeout`
- `Cloudflare ... Error 1020` (blocked by rules)

### Browser / runtime
- `browserType.launch`
- `Browser has been closed`
- `Target page, context or browser has been closed`
- `Navigation timeout` (usually — see caveat below)

### Caveat: Navigation timeout
`Navigation timeout` on a single page after the app deploys is usually REGRESSION (slow or broken page). `Navigation timeout` across multiple unrelated pages in the same run is usually ENVIRONMENT (infra-wide slowdown). Check the scope before classifying.

---

## 4. Error-pattern → classification quick lookup

| Error pattern | Likely classification | Note |
|---------------|----------------------|------|
| `Element not found` / `locator.click: Target closed` | REGRESSION | UI changed, selector broken |
| `Timeout X ms exceeded waiting for locator` | REGRESSION or FLAKY | Check history |
| `expect(received).toBe(expected)` — assertion mismatch | REGRESSION | Logic or data change |
| `status code 500` / `status code 502` | ENVIRONMENT or REGRESSION | Isolated → REGRESSION; many tests → ENV |
| `ECONNREFUSED` | ENVIRONMENT | App down or port wrong |
| `Navigation timeout` on one page after deploy | REGRESSION | Slow page |
| `Navigation timeout` across many tests | ENVIRONMENT | Infra slowdown |
| Intermittent pass/fail on unchanged build | FLAKY | Same build, different outcomes |
| `Test passed in run #N`, fails in #N+1 with code change between | REGRESSION | Bisect the diff |
| `SyntaxError` / `TypeError` in test file | REGRESSION (test code) | Test code itself broken |
| `No snapshot found` | NEW TEST FAILURE | First run of a visual snapshot |

---

## 5. Computing flakiness (the >20% rule)

A test is FLAKY if its failure rate over the last N runs on unchanged application builds exceeds 20%.

### Algorithm

```
1. Gather results for this test from the last 10 runs of the same workflow + environment.
   (If < 10 available, use what exists but require at least 5.)

2. Exclude runs that were against a different application build
   (different commit SHA on the main branch between runs, or different deploy).
   The goal is "same build, different outcomes" — the signature of flakiness.

3. Among the remaining runs:
   failure_rate = failed_count / total_count

4. Apply threshold:
   - failure_rate > 0.20 → FLAKY
   - 0 < failure_rate ≤ 0.20 → REGRESSION candidate (unless only-in-last-run, then new failure)
   - failure_rate == 0 → not applicable (test is passing — why are you here?)

5. If N < 5: output "INSUFFICIENT HISTORY" — do not guess.
```

### Retry-aware flakiness (conscious-divergence projects only)

This repo ships `retries: 0` everywhere (`playwright.config.ts` — deterministic tests; a retry hides the flake). With the shipped config a retry-pass signal **cannot occur**: every flake surfaces as a plain failure and is caught by the >20% rule above. Skip this subsection unless your project has consciously diverged.

If a downstream project has deliberately enabled retries (see the "Conscious divergence: enabling retries" box in `ci-cd-integration.md`), a test that passes on retry is a hidden flake. Check Allure `retriesCount` for each result — a `passed` result with `retriesCount > 0` counts as a flake observation even though the final status is green.

When reporting flakiness rate on such a project, include retry-passes in the numerator:
```
effective_failure_rate = (failed + retried_passes) / total
```

### Getting history with gh CLI

```bash
# Last 10 runs
gh run list --workflow=regression.yml --limit=10 --json databaseId,conclusion,createdAt,headSha

# Pair of adjacent runs for comparison
gh run list --workflow=regression.yml --limit=2 --json databaseId,headSha -q '.[] | "\(.databaseId) \(.headSha)"'

# Download artifacts for trend analysis
for RUN_ID in $(gh run list --workflow=regression.yml --limit=10 --json databaseId -q '.[].databaseId'); do
  gh run download $RUN_ID -n merged-allure-results-staging -D ./history/$RUN_ID/ 2>/dev/null || true
done
```

---

## 6. REGRESSION vs FLAKY — the hard cases

The decision is easy when history is clean. The hard cases:

### Case 1: First failure after a green streak
Test passed 5 runs in a row, now fails once.
- If a code change deployed between the last pass and this failure → REGRESSION.
- If no deploy, same commit → FLAKY candidate (monitor next run).

### Case 2: Intermittent pattern matches release cadence
Test fails every Monday morning but passes other days.
- Look at scheduled jobs, cron, maintenance windows → probably ENVIRONMENT (infra cycle).

### Case 3: Passes on retry consistently (retry-enabled projects only)
Every run shows `failed → passed on retry` — only possible on a project that consciously diverged from the shipped `retries: 0`.
- This is FLAKY. The user sees green but the underlying test is unstable. Stabilize it.

### Case 4: One assertion flakes within a test with N assertions
The same assertion in a multi-assertion test fails intermittently; others always pass.
- The test itself is FLAKY. Do not mark the entire test REGRESSION. Fix the one assertion (usually a timing issue).

### Case 5: Fails on one browser only
Test passes on chromium + firefox, fails on webkit.
- If the feature uses browser-specific APIs → REGRESSION (browser compat broken).
- If it is visual or layout-sensitive → could be FLAKY on webkit's slower render. Check history per browser.

---

## 7. Severity (orthogonal to classification)

Severity = business impact. A FLAKY test on checkout is still CRITICAL severity.

| Severity | Criteria |
|----------|----------|
| CRITICAL | Core user journey (login, signup, checkout, payment, core search). Any test tagged `@critical`. |
| HIGH | Major feature area (profile, dashboard, primary search filters, account management) |
| MEDIUM | Secondary feature (sorting, preferences, non-primary filters, notifications) |
| LOW | Admin-only, edge case, rare scenario, internal tools |

Severity inputs:
1. Test tags (`@critical`, `@smoke`, `@regression`).
2. Suite name (Auth, Booking, Payment = CRITICAL; Admin, Settings = LOW/MEDIUM).
3. TMS ticket priority if linked.

### Release impact matrix (classification × severity)

|                | CRITICAL | HIGH | MEDIUM | LOW |
|----------------|----------|------|--------|-----|
| **REGRESSION** | NO-GO | NO-GO | CAUTION | CAUTION |
| **FLAKY**      | CAUTION + stabilize now | CAUTION | monitor | monitor |
| **KNOWN-BLOCKED** | excluded from gating — reassess `{BUG-KEY}` urgency | excluded, document | excluded, document | excluded, document |
| **KNOWN**      | CAUTION (reassess ticket) | document | document | document |
| **ENV**        | re-run, escalate infra | re-run | re-run | re-run |
| **NEW TEST**   | verify manually before GO | verify | verify | verify |

---

## 8. Classification report sections

Every classified failure needs this evidence block, regardless of category:

```markdown
#### {test_name}
- Test ID: {atc_id}
- Suite: {suite}
- Classification: {KNOWN-BLOCKED / REGRESSION / FLAKY / KNOWN / ENVIRONMENT / NEW}
- Severity: {CRITICAL / HIGH / MEDIUM / LOW}
- Run: {run_url}
- Last passed: {date} (run #{run})
- Error:
  ```
  {full_error_message}
  ```
- Probable cause: {one-paragraph analysis}
- Screenshot: {path or url}
- Trace: {path or url}
- Ticket: {created-or-linked-ticket}
```

---

## 9. Anti-patterns

- **Classifying without reading the error.** The error text is usually definitive. Never classify based on test name alone.
- **Calling everything "flaky".** Flaky is a last resort after ruling out REGRESSION and ENVIRONMENT. Lazy flaky-tagging hides real regressions.
- **Calling everything "environment".** If only one test in the suite fails with a network error and that endpoint is hit successfully by other tests in the same run, it is not environment — it is a bug in the code path that test exercises.
- **Marking NEW TEST as REGRESSION.** A first-run failure has no prior pass to regress from. Mark as NEW TEST, verify manually, then reclassify.
- **Ignoring retry-passes (retry-enabled projects only).** With the shipped `retries: 0` there are no retry-passes to ignore. On a project that consciously enabled retries, a test that passes on retry is unstable — surface it in the flaky bucket even though Allure shows green.
- **Guessing flakiness with < 5 runs of history.** Not enough data — mark as "insufficient history" and revisit.

---

## 10. Classification flow summary (for report footer)

```
Total failed: X
├── KNOWN-BLOCKED:  X  ← parked behind {BUG-KEY}, excluded from gating pass-rate
├── REGRESSION:     X  ← release blockers (sorted by severity DESC)
├── FLAKY:          X  ← schedule stabilization
├── KNOWN ISSUE:    X  ← documented, not blocking
├── ENVIRONMENT:    X  ← re-run after infra check
└── NEW TEST:       X  ← manual verification required
```

# GitHub Pages Setup — publish Allure reports for THIS project

> One-time setup that makes the Allure reports your CI already pushes to
> `gh-pages` actually reachable in a browser. The regression / smoke / sanity
> workflows shipped with this boilerplate publish to the `gh-pages` branch out
> of the box via `scripts/ci/publish-allure-pages.ts` (Allure 3, same
> `allurerc.mjs` as local runs: the Awesome report served directly with its
> Report | Graphs | Timeline modes, trend history per env/suite, latest-run
> redirect, last-10-runs retention) — but GitHub does NOT serve that branch
> until Pages is explicitly enabled on the repo. This reference is the full
> maneuver, learned the hard way on the boilerplate repo itself.

## When to run this

- User asks any variant of: "set up GitHub Pages", "publish the Allure
  reports", "why is the report URL 404", "enable the reports site".
- `/adapt-framework` finished wiring CI and the project wants browsable
  reports.
- The `Post Report URL` step of a suite run prints a URL that returns 404.

## Step 0 — Allure version-currency check (MANDATORY)

Before touching Pages, run the **§Allure version-currency check** from
`SKILL.md` §Local reporting: the boilerplate's pinned `allure` /
`allure-playwright` / `allure-js-commons` versions are usually behind upstream
by the time a scaffolded project reaches this setup. Summarize the news to the
user, offer the same-major `bun update`, never cross a major silently, and
re-generate one report to confirm the root `index.html` still opens the
Awesome report directly (Report | Graphs | Timeline). The published site
inherits whatever version generates in CI, so an outdated local pin means an
outdated public report.

## Preconditions (probe, don't assume)

| Check | Command | Blocker if |
| --- | --- | --- |
| gh CLI authenticated | `gh auth status` | fails → stop, ask user to `gh auth login` |
| gh-pages branch exists on origin | `git ls-remote --heads origin gh-pages` | empty → run a suite first (any Allure deploy creates it) |
| Repo visibility | `gh repo view --json visibility` | `PRIVATE` + free plan → Pages requires a paid plan; surface to user |
| Pages state | `gh api repos/{owner}/{repo}/pages` | `404` means NOT enabled → proceed with setup |

## Step 1 — Enable Pages serving the gh-pages branch

```bash
gh api -X POST repos/{owner}/{repo}/pages \
  -f "source[branch]=gh-pages" -f "source[path]=/"
```

Success returns `"html_url": "https://{owner}.github.io/{repo}/"`. If it
returns `409 Conflict`, Pages is already enabled — check its source with
`gh api repos/{owner}/{repo}/pages` and fix the branch if it points elsewhere
(`gh api -X PUT ... -f "source[branch]=gh-pages"`).

## Step 2 — Verify the first build (KNOWN GOTCHA: it can silently fail)

The first build after enablement — on a branch that already has content — can
error with a generic "Page build failed." or hang in `building` for a long
time. Do NOT push commits to "wake it up"; request a rebuild instead:

```bash
# status of the latest build
gh api repos/{owner}/{repo}/pages/builds/latest --jq '{status, error: .error.message}'

# if status is "errored" (or stuck "building" for >10 min): force a rebuild
gh api -X POST repos/{owner}/{repo}/pages/builds
```

Then poll `builds/latest` until `status: built` and confirm with curl:

```bash
curl -s -o /dev/null -w "%{http_code}" https://{owner}.github.io/{repo}/{env}/regression/
```

`{env}` is the project's active environment (e.g. `staging`). Expect `200`.

## Step 3 — Long-term storage hygiene (prevent unbounded branch growth)

Two independent growth vectors, two controls:

1. **Working tree** — already controlled: `scripts/ci/publish-allure-pages.ts`
   prunes to the last 10 run dirs per env/suite (`--keep`, adjustable in each
   workflow). Screenshots/videos live inside each report and rotate with it;
   `history.jsonl` (trend data) persists independently and stays small.
2. **Git history** — NOT controlled by run pruning: every deploy commit keeps
   its blobs in history forever, so the branch grows on every run even though
   the served site does not. Fix: a scheduled squash job that rewrites
   `gh-pages` to a single orphan commit holding the current content. This is
   the ONLY sanctioned force-push in the project — `gh-pages` is 100%
   workflow-generated, nobody develops on it. Template (adapt repo name; runs
   monthly + on demand): see `.github/workflows/pages-squash.yml` in the
   boilerplate source repo (upex-galaxy/agentic-qa-boilerplate) — copy it
   verbatim; it is repo-agnostic.

## Report to the user

- Site URL + the per-suite report URLs (`/{env}/regression/`, `/{env}/smoke/`,
  `/{env}/sanity/` — whichever workflows the project runs).
- Whether the squash job was installed (recommended: yes).
- Reminder: on public repos the reports are public — screenshots may leak
  UI/data of the app under test; confirm the team is OK with that or keep the
  repo private (Pages on private repos needs a paid plan).
- If the team needs reports behind a login instead, switch to the PRIVATE
  Test Report Portal mode — runbook: `references/private-hosting-setup.md`
  (the suite workflows are already dual-mode; only secrets are wired).

## Scope guard

This reference sets up ALLURE report publishing for consumer projects. The
boilerplate's own `pages.yml` (docs hub: KATA Academy + decks + homepage)
is boilerplate-EXCLUSIVE — never replicate it in a consumer project; the
scaffolder deliberately excludes it.

# Private Report Hosting — AI-Executed Setup Protocol

> **Purpose**: switch a project's CI Allure reports from public GitHub Pages to
> the PRIVATE, auth-walled **Test Report Portal**
> ([upex-test-report-portal](https://github.com/upex-galaxy/upex-test-report-portal),
> v2). Reports become reachable only after login, bytes live in a private
> Cloudflare R2 bucket, trends/retention keep working, and the portal indexes
> every run by environment/strategy.
>
> **Execution model**: THE AI RUNS THIS SETUP. Every step below is a command
> the AI executes, except the blocks marked `HUMAN CHECKPOINT` — those are the
> only actions that require the human (account sign-ups, one-time auth
> handshakes, and the optional OAuth app, which has no API). Announce each
> checkpoint, wait for the human, verify, continue.
>
> **When to use**: "reports must be private", "no publiques evidencia
> pública", "protege los reportes con login". If the org has **GitHub
> Enterprise Cloud**, offer the zero-infra shortcut first (repo Settings →
> Pages → visibility Private).

## Architecture (what you are wiring)

```
CI (this repo)                                  Portal (deployed once per org)
  1. tests -> allure-results                      Vercel (Next.js + NextAuth)
  2. GET history from portal        ------->      Supabase Postgres (runs index)
  3. bunx allure generate                         Cloudflare R2 PRIVATE bucket
  4. aws s3 sync -> R2 (direct)     ------->        {project}/{env}/{suite}/{run}/
  5. PUT history + POST /api/runs   ------->        {project}/{env}/{suite}/history.jsonl
                                                  Viewer: login -> /api/view proxy streams bytes
```

- The publish step in `regression.yml` / `smoke.yml` / `sanity.yml` is already
  dual-mode: portal **iff the `PORTAL_URL` secret exists**, else public Pages.
- Publisher: `scripts/ci/publish-allure-portal.ts` (synced downstream by
  `bun run update`). Retention is server-side (portal cron).
- In portal mode gh-pages is unused: after verification, disable Pages
  serving (Settings → Pages → Source: None); deleting the gh-pages branch
  requires explicit user confirmation (Critical Rule #6).

## Part 0 — Already-configured detection (idempotency gate, run FIRST)

Probe before provisioning anything — every level short-circuits:

```bash
gh secret list | grep -E "PORTAL_URL|PORTAL_PROJECT|PORTAL_API_KEY|R2_"   # repo already wired?
```

| Probe result | Meaning | Action |
|---|---|---|
| All 7 secrets present | **This repo is fully wired** | Nothing to install. Offer: verify (Part C), rotate key, or change retention. |
| Some secrets present | Partial/broken wiring | Diff against the Part B table, fill only the missing ones. |
| No secrets, but user/org has a portal (ask; also check Engram `mem_search "portal URL"`) | Part A done previously | `curl -s -o /dev/null -w "%{http_code}" <PORTAL_URL>/api/metrics` → `401` = portal alive and walled → skip to Part B. |
| No secrets, no portal | Fresh install | Run Part A → Part B → Part C. |

Part A steps are themselves check-before-create: `supabase projects list`
before `projects create`, `wrangler r2 bucket list` before `bucket create`,
`vercel ls` before `vercel link`, and `create-project.ts` upserts (re-running
it ROTATES the project's API key — only do that deliberately, it invalidates
the old key in CI secrets).

## Part A — Portal deployment (once per organization)

Skip to Part B if the org already runs a portal instance (ask for its URL).
Otherwise, **FORK the portal** (not a plain clone): the org gets its OWN
repo — Vercel connects to it (every push auto-deploys), branding/tweaks have
a home, and upstream updates arrive via the `upstream` remote that `gh` wires
automatically:

```bash
gh repo fork upex-galaxy/upex-test-report-portal --clone -- ../test-report-portal
cd ../test-report-portal && bun install
# Later upgrades: git pull upstream main && git push   (auto-redeploys)
```

The local checkout is the deploy vehicle for the org's own infrastructure —
after setup it is only revisited for upgrades and `create-project` runs.
(Humans without an AI can use the "Deploy to Vercel" button in the portal
README instead — same fork-and-deploy result in one click.)

### A0 — HUMAN CHECKPOINT: accounts + three one-time credentials

Ask the human for (offer `! <command>` where interactive):

1. **Supabase Personal Access Token** — dashboard → account → Access Tokens →
   generate. Or run `! supabase login` (browser handshake). Export as
   `SUPABASE_ACCESS_TOKEN`.
2. **Cloudflare**: account with **R2 enabled** (R2 → activate; asks for a
   payment method even though the free 10 GB tier is $0) + a **user API token**
   with permission **API Tokens: Edit** (dashboard → My Profile → API Tokens).
   Export as `CF_MASTER_TOKEN`. Alternative: `! wrangler login`.
3. **Vercel**: `! vercel login` (or a token from vercel.com/account/tokens →
   export `VERCEL_TOKEN`).

Never echo these values into logs, commits, or files.

### A1 — Supabase project + schema (AI)

```bash
supabase orgs list                                   # get org id
supabase projects create test-report-portal \
  --org-id <ORG_ID> --region <closest> \
  --db-password "$(openssl rand -hex 16)"            # record ref from output
supabase link --project-ref <REF>
supabase db push                                     # applies supabase/migrations/002_v2_schema.sql
supabase projects api-keys --project-ref <REF>       # collect keys
```

Collect: `NEXT_PUBLIC_SUPABASE_URL=https://<REF>.supabase.co`, the
**publishable** key (`sb_publishable_...` → `NEXT_PUBLIC_SUPABASE_ANON_KEY`)
and the **secret** key (`sb_secret_...` → `SUPABASE_SERVICE_KEY`). New
Supabase projects no longer issue legacy `anon`/`service_role` JWTs — the
`sb_*` keys are drop-in replacements.

### A2 — R2 bucket + scoped S3 credentials (AI)

```bash
bunx wrangler r2 bucket create test-reports          # keep public access OFF (default)
```

Create the bucket-scoped S3 credentials **via API** (no dashboard needed):
`POST https://api.cloudflare.com/client/v4/user/tokens` with
`Authorization: Bearer $CF_MASTER_TOKEN` and an R2 Object Read & Write policy
scoped to the bucket (see
developers.cloudflare.com/r2/api/tokens → "Create API tokens via API").
Then derive:

- `R2_ACCESS_KEY_ID` = the created token's `id`
- `R2_SECRET_ACCESS_KEY` = `echo -n "<token value>" | shasum -a 256` (hex)
- `R2_ACCOUNT_ID` = `bunx wrangler whoami` account id · `R2_BUCKET` = bucket name

Sanity-check before continuing:

```bash
AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY AWS_DEFAULT_REGION=auto \
aws s3 ls "s3://$R2_BUCKET/" --endpoint-url "https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com"
```

### A3 — HUMAN CHECKPOINT (optional): OAuth app

Google/GitHub OAuth clients **cannot be created via any API or CLI** — this
is the one irreducible dashboard step (~5 min). The portal works WITHOUT it
(admin credentials login), so offer both paths:

- Skip now → deploy with credentials-only login; add OAuth later.
- Do now → Google Cloud console → OAuth client (Web), redirect URI
  `https://<portal-domain>/api/auth/callback/google` → human hands over
  `GOOGLE_ID` + `GOOGLE_SECRET`.

### A4 — Vercel deploy (AI)

From the portal repo clone (CLI shown; the Vercel MCP is an equivalent path):

```bash
vercel link --yes                                    # create/link the project
# Set every env var (production): loop `vercel env add <NAME> production`
#   NEXTAUTH_URL=https://<assigned-domain>   NEXTAUTH_SECRET=$(openssl rand -hex 32)
#   LOGIN_EMAIL / LOGIN_PASSWORD (bcrypt: bun scripts/generate-password-hash.ts)
#   AUTHORIZED_EMAIL_DOMAINS=<company.com>   AUTHORIZED_EMAILS=
#   GOOGLE_ID / GOOGLE_SECRET (if A3 done)
#   Supabase trio (A1) · R2 quartet (A2) · CRON_SECRET=$(openssl rand -hex 24)
#   NEXT_PUBLIC_APP_URL=https://<assigned-domain>
vercel deploy --prod --yes
```

The retention cron registers automatically from `vercel.json`.

### A5 — Wall verification (AI, mandatory)

```bash
curl -s -o /dev/null -w "%{http_code}" https://<portal>/api/metrics   # expect 401
```

Then with browser automation (or ask the human for an incognito check):
portal root → login page renders; login with admin credentials → dashboard.

## Part B — Per-project wiring (each downstream repo) (AI)

1. **Provision the project** (in the portal repo clone, Supabase envs loaded):

   ```bash
   bun scripts/create-project.ts <project-slug> "<Display Name>" --retention-runs 30
   ```

   Capture the one-time API key from stdout — transfer it ONLY into the
   secret below, never into a file or log.

2. **Secrets** in the consuming repo:

   ```bash
   gh secret set PORTAL_URL --body "https://<portal-domain>"
   gh secret set PORTAL_PROJECT --body "<project-slug>"
   gh secret set PORTAL_API_KEY --body "<one-time key>"
   gh secret set R2_ACCOUNT_ID --body "..."     # + R2_ACCESS_KEY_ID,
   gh secret set R2_BUCKET --body "..."         #   R2_SECRET_ACCESS_KEY
   ```

3. **Trigger a suite** (`gh workflow run smoke.yml`) and watch the publish
   step: history GET → generate → s3 sync → history PUT → run registered.

   No CI available yet? Run the manual publish test instead — portal repo
   `SETUP.md` §6.5 has the copy-paste flow with any local `allure-report/`.

## Part C — Verification checklist (MANDATORY before declaring done)

- [ ] CI publish step green; job summary shows the portal `viewUrl`.
- [ ] `viewUrl` logged in → report renders (iframe via `/api/view/...`).
- [ ] `viewUrl` + a direct asset URL in incognito/curl → login wall / 401,
      never bytes.
- [ ] Second run on the same stream → trend charts render (history works).
- [ ] Portal dashboard lists the run under the right environment/strategy.
- [ ] If migrating from public Pages: Pages serving disabled after user
      confirms.

## Failure modes

| Symptom | Cause | Fix |
|---|---|---|
| `Missing required environment variable: PORTAL_*` in CI | Secret not set | Part B step 2 |
| `401 Invalid credentials` on history/runs | Wrong `PORTAL_API_KEY` / slug mismatch | Re-run create-project (rotates key), update secret |
| `400 reportPrefix must be ...` | Publisher/portal contract drift | `bun run update` in the consuming repo |
| Report iframe 404s assets | Sync hit wrong bucket/prefix | Check `R2_BUCKET` secret + CI log line `r2://...` |
| `aws s3 ls` in A2 fails with 403 | Token policy not scoped to the bucket / wrong secret derivation | Recreate token; secret = SHA-256 of the token VALUE, not the id |
| OAuth login rejects a valid teammate | Domain missing in `AUTHORIZED_EMAIL_DOMAINS` | Add + redeploy |
| Trends empty after 2+ runs | History PUT failing | Check publish log; key rotated mid-stream? |
| Cron never runs | `CRON_SECRET` unset in Vercel | Set + redeploy |

## Boundaries

- NEVER print `PORTAL_API_KEY`, R2 secrets, or Supabase secret keys into
  logs, commits, Jira, or chat transcripts beyond what the human must copy.
- The publisher never deletes from R2; only the portal cron does.
- One R2 bucket serves ALL projects — isolation is enforced by the portal
  (API key ↔ slug ↔ prefix validation).
- Human-only steps are ONLY: account sign-ups, the three A0 handshakes, R2
  payment-method activation, and the optional A3 OAuth app. Everything else
  is yours to execute.