Claude Skill

tiered-rubric-loop

Drops a combined deterministic and semantic rubric verification loop into any project, acting as the loop component of an agentic harness. Automatically leverages existing Spec-Driven Development (SDD) and Test-Driven Development (TDD) artefacts - or creates lightweight ones inli

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download derailed-dash-dazbo-agent-skills-skills_tiered-rubric-loop-44f16ae.zip · 20 KB

Install

skills CLI npx skills add https://github.com/derailed-dash/dazbo-agent-skills/tree/main/skills/tiered-rubric-loop
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install derailed-dash-dazbo-agent-skills@llmmart
Git git clone https://github.com/derailed-dash/dazbo-agent-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole derailed-dash/dazbo-agent-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Tiered Rubric Loop (tiered-rubric-loop)

A fast, loosely coupled way to drop a combined deterministic + semantic verification loop into any repository as the verification engine of an agentic harness.

It automatically reuses any Spec-Driven Development (SDD) (docs/SPEC.md or SPEC.md) and Test-Driven Development (TDD) artefacts already in the repo, or bootstraps a lightweight B1–Bn binary rubric checklist inline if none exist.

Instead of waking an expensive reasoning subagent on every single pass, or trusting a zero-thinking classifier blindly on deep security and concurrency invariants, each assertion flows through three escalating stages:

  1. Tier 1 — Deterministic Checks (fast): Runs local linters, typecheckers, and unit tests (make check, codespell, ruff, pyright, pytest) first. Any failure stops the pass immediately so you never waste API calls on broken code.
  2. Tier 1.5 — Fast Semantic Scoring (fast model, no output tokens): Evaluates open binary (yes/no) rubric criteria (B1–Bn) in a single prefill pass using the Gemini Decisions API (gemini-decision-1 via scripts/eval_rubric.py). Straightforward editorial and doc-sync checks pass immediately, while obvious omissions fail fast without waking a subagent.
  3. Tier 2 — Stateless Semantic Critique: Spawns a fresh, read-only reviewer subagent only when Tier 1.5 is uncertain ("grey zone"), or for complex architectural and security invariants ([arch]) that need deeper multi-file reasoning.

Table of Contents


How the 3-Stage Funnel Works

Tier Mechanism Latency & Cost What It Owns
Tier 1 (Deterministic) Local linters, typecheckers, unit tests (make check, codespell, ruff, pyright, pytest) Seconds, $0 Syntax, static types, formatting, and deterministic test execution. Any failure short-circuits immediately before Tier 1.5 or Tier 2 runs.
Tier 1.5 (Calibrated Probabilistic) gemini-decision-1 via client.decisions.create (v1beta) in scripts/eval_rubric.py ~150–250 ms, $0.10 / 1M input tokens, 0 output / thinking tokens Evaluates up to 1,000 closed-ended noul (yes/no) assertions over a single shared diff/spec/media payload. Terminates clear mechanical passes (PASS), rejects obvious omissions immediately (FAIL_FAST), and flags uncertain items for L2.
Tier 2 (Stateless Semantic Critic) Fresh read-only flash subagent with zero prior coding chain-of-thought 10–40 s, standard model tokens Diagnoses grey-zone escalations and performs one final L2 sign-off on [arch] items after Tier 1.5 is green.

Key safety rule: Because gemini-decision-1 runs in a single prefill forward pass with zero thinking tokens and zero file-reading tools, it can produce false positives on subtle logic, race conditions, or auth flaws. Therefore, Tier 1.5 is terminal for mechanical/editorial checks, fail-fast on clear omissions, and provisional for [arch] assertions.


Plain-Markdown B1–Bn Contract & Tagging

The sole shared contract is a Markdown table or bullet list of numbered binary assertions (B1–Bn) in docs/SPEC.md, SPEC.md, or LOOPS.md (with optional --spec <path> override):

| ID | Falsifiable Condition | Verification Target |
| :--- | :--- | :--- |
| `B1 [det]` | Static gates (`codespell`, `ruff`, `pyright`) and offline tests exit `0` | `make check` |
| `B2` | `.env.template` and `README.md` document every newly added variable and flag | Doc review |
| `B3 [arch]` | Auth middleware rejects expired tokens and never retries `4xx` responses | `tests/unit/test_auth.py` |

Other supported rubric formats and tags:

  • Bullet assertions: - **B4**: CLI help panel renders in English (UK)
  • Atomic sub-IDs: B4a, B11b
  • Numbered items: 1. `B5 [docs]`: ...
  • Task checkboxes: - [ ] **B6 [concurrency]**: ... or - [x] **B6**: ...
  • Extended tags: [mech], [docs], [ui], [visual], [perf], [concurrency] (unrecognised tags warn on stderr and default safely to [mech] instead of silently dropping the assertion)
Tag Meaning Tier 1.5 Handling
[det] (or deterministic Verification Target such as make check, pytest tests/..., codespell -s, ruff check .) Already proven by Tier 1 shell exit codes Skipped by Tier 1.5 by default (--include-det overrides) so the model never guesses whether pytest passed from a text diff.
(untagged, [mech], [docs], [ui], [visual], [multimodal]) Mechanical, editorial, doc-sync, schema, or UI convention check Terminal PASS when P(yes) >= 0.90, confidence >= 0.80, and no evidence was truncated; FAIL_FAST when <= 0.20; otherwise ESCALATE_L2_FLASH.
[arch] (also [security], [logic], [concurrency], [perf]) Concurrency, security, state invariant, performance, or test-realism check Provisional pass (PROVISIONAL_ARCH_PASS) at Tier 1.5; gated for one final Tier 2 (Model: "flash") review once all items clear Tier 1.5.

Repo-relative file paths and local images named in Verification Target or inline backticks are automatically resolved and attached (blocking .env, secret env files, *.enc, and paths outside --repo-root; pass --no-auto-target-files to disable).

Writing High-Signal B1–Bn Assertions

  • Keep assertions atomic: Test one falsifiable invariant per Bi rather than bundling three functions or eight files into one question—single-invariant assertions give sharp P(yes) separation and trigger FAIL_FAST accurately on regressions.
  • Name concrete paths in Verification Target: Prefer src/auth.py, tests/unit/test_auth.py over generic Doc review / Code review so eval_rubric.py auto-attaches untouched context files even in narrow working-tree diffs.
  • Separate canonical registries from summary copy: Don't assert identical title formatting across both machine registries (navigation.yaml) and summarised overview pages (README.md, spotlight cards) unless exact parity is truly required.

Loose Coupling & Companion Skills Matrix

tiered-rubric-loop requires zero hard dependencies on other skills and degrades cleanly when optional companions or API keys are absent:

Companion Skill / Dependency When Present Standalone Fallback (When Absent)
maintaining-core-documentation / spec-driven-development Reads B1–Bn directly from docs/SPEC.md or SPEC.md. Reads any local SPEC.md / LOOPS.md, or asks you to agree a compact B1–Bn list inline before coding.
test-driven-development Delegates the Act step to full Red-Green-Refactor execution. Enforces the core TDD rule inline: write/update a failing test first, change one semantic behaviour per pass, never weaken assertions.
loopy Supplies the outer Observe -> Choose -> Act -> Verify -> Record -> Stop governance and LOOPS.md receipts. Runs the identical bounded loop directly with the receipt format below.
local-code-review Stateless Tier 2 read-only subagent runs local-code-review on escalated/[arch] items. Uses the built-in Stateless Tier 2 Critic Prompt below.
GEMINI_API_KEY / GOOGLE_API_KEY Enables ~200 ms Tier 1.5 screening via eval_rubric.py. When eval_rubric.py exits 3 (missing/invalid key), skips Tier 1.5 cleanly and executes Tier 1 -> Tier 2 directly while noting how to enable Tier 1.5.

Bounded Loop Workflow

Track every pass around Observe -> Choose -> Act -> Verify -> Record -> Stop:

  1. Observe: Read the specification (docs/SPEC.md, SPEC.md, or LOOPS.md), git status --short, and current failing tests/checks. Confirm B1–Bn exist (draft and confirm them first if missing).
  2. Choose: Pick one highest-value unverified semantic behaviour or failing assertion for this pass (mechanical lint/spelling fixes may be batched).
  3. Act (TDD): Write or update the failing test first (Red), confirm it fails, then implement the minimal clean change (Green) and refactor without weakening tests.
  4. Verify (3-Stage Funnel):
    • Tier 1: Run deterministic checks (make check or repo lint + test commands). If Tier 1 fails, stop verification immediately and return to Choose/Act—do not spend tokens on Tier 1.5 or Tier 2.
    • Tier 1.5: Run eval_rubric.py --json over the spec, diff, and touched files (plus --image / --pdf for diagram/UI specs).
    • Tier 2: Escalate only grey-zone items immediately, or run a single final [arch] critic pass once Tier 1.5 reports zero FAIL_FAST and zero grey-zone items.
  5. Record: Append pass outcome, ratchet_score (mean P(yes)), and remaining failures to LOOPS.md (or session progress notes).
  6. Repeat or Stop: Enforce the Circuit Breaker Trio before starting another pass.

Running Tier 1.5 (eval_rubric.py)

Resolve scripts/eval_rubric.py relative to where tiered-rubric-loop is installed:

  • Global Antigravity path: ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py
  • Project-local path: .agents/skills/tiered-rubric-loop/scripts/eval_rubric.py

In sandboxed agent runtimes that block outbound HTTPS by default, invoke uv run with unsandboxed network permission (BypassSandbox: true in Antigravity) for live API calls, or pass --offline for sandboxed --dry-run checks.

# 1. Working-tree loop: auto-discover SPEC.md / docs/SPEC.md / LOOPS.md
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --json

# 2. Branch or merged PR gating: evaluate committed changes against base ref / range
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --base main --json
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
  --git-range origin/main...HEAD \
  --only B3,B4 \
  --json

# 3. Explicit spec, scoped files, and multimodal diagram/UI screenshot check
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
  --spec docs/SPEC.md \
  --file src/auth.py \
  --image docs/images/architecture.png \
  --json

# 4. Offline dry-run inside sandbox (verifies rubric, targets, and truncation zero-API)
uv run --offline ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --dry-run --json

For large PRs (> 8 files or > 100 KB), shard evaluations by assertion group via --only B3,B4 so files stay within --max-file-chars (24,000) and --max-evidence-chars (180,000). If any file or combined payload is truncated, eval_rubric.py populates truncated_sources and downgrades mechanical PASS to ESCALATE_L2_FLASH unless --allow-truncated-pass is explicitly set.

Exit Codes

Code Meaning Loop Action
0 Tier 1.5 clear (all evaluated items are PASS or PROVISIONAL_ARCH_PASS) If final_l2_arch_ids is non-empty, run one Tier 2 (Model: "flash") critic pass on those IDs; otherwise exit loop as Success.
1 One or more FAIL_FAST or ESCALATE_L2_* items (or API call error) Address FAIL_FAST items directly; escalate grey-zone items (ESCALATE_L2_*) only when root cause is non-obvious.
2 Usage error, missing spec, or no B1–Bn assertions parsed Fix spec path or rubric formatting before continuing.
3 Missing/invalid GEMINI_API_KEY or GOOGLE_API_KEY Degrade gracefully to Tier 1 -> Tier 2 and inform the user once.

Threshold Routing & Stateless Tier 2 Critic

Default calibrated thresholds (overridable via --pass-threshold, --fail-threshold, --min-confidence):

  • PASS (P(yes) >= 0.90, confidence >= 0.80, mechanical): Terminal pass. No L2 model runs.
  • PROVISIONAL_ARCH_PASS (P(yes) >= 0.90, confidence >= 0.80, [arch]): Held until all other assertions pass, then verified once by a Tier 2 (Model: "flash") critic.
  • FAIL_FAST (P(yes) <= 0.20, confidence >= 0.80): Returned directly to the coding agent—do not burn an L2 turn repeating an obvious omission.
  • ESCALATE_L2_FLASH (0.20 < P(yes) < 0.90 or confidence < 0.80): Escalate only those specific Bi IDs to a fresh read-only subagent.

Subagent Routing Across Coding Agents

Always spawn Tier 2 as a fresh, stateless, read-only subagent that never sees the coding agent's prior chain-of-thought or failed attempts. When spawning, use a fast or flash agent. For example, in Antigravity, call invoke_subagent with TypeName: "research" and Model: "flash".

Built-In Stateless Tier 2 Critic Prompt

Pass this exact prompt (filling in only the target Bi IDs, spec path, and changed files):

You are a stateless, read-only Tier 2 semantic critic. You have NOT seen the
coding session and must judge solely from the specification and actual files on disk.

Evaluate ONLY these rubric assertions: [INSERT B_i IDs AND CONDITIONS].
Read: [SPEC.md / docs/SPEC.md] and inspect the changed/untracked files + git diff.

Rules:
1. Cite exact file paths, line numbers, and observed code/test behaviour.
2. Check whether tests verify real behaviour rather than mocking away logic.
3. For each assertion ID, return PASS or FAIL with a concise root-cause note.
4. Do not invent fixes or assume uninspected files pass.

Circuit Breaker Trio & Continuous Ratchet

Stop the loop whenever any of these three conditions triggers:

  1. 100% Pass: Tier 1 passes, Tier 1.5 reports exit_code == 0, and any final_l2_arch_ids clear their final Tier 2 (Model: "flash") review (Success).
  2. Pass Budget Exhausted: Maximum 4–5 passes per slice reached (Exhausted—escalate remaining blockers to the user).
  3. 2 Consecutive Non-Improving Passes: Neither failing assertion count decreases nor ratchet_score (mean P(yes) across evaluated B1–Bn items) improves across two consecutive passes (No progress—escalate to the user rather than spinning).

Tracking the continuous ratchet_score avoids premature stalls when a pass genuinely moves an assertion from e.g. P(yes) = 0.25 to 0.82 while still short of 0.90, and catches silent regressions immediately.


Loop Receipt Format (LOOPS.md)

Record completed runs (or append per-pass telemetry) concisely:

## Tiered Rubric Loop Receipt

- **Target**: [Slice / Feature / PR]
- **Spec**: [docs/SPEC.md] (`B1`–`Bn`)
- **Passes**: [N / 5] | **Outcome**: `Success` | `Exhausted` | `No progress` | `Blocked`
- **Tier 1**: `make check` -> PASS
- **Tier 1.5 Ratchet (`gemini-decision-1`)**: `0.58 -> 0.94 -> 0.98` (skipped `[det]`: `B1`)
- **Tier 2 (`Model: "flash"`)**: Verified `[arch]` IDs `B3` -> PASS
- **Remaining / Escalation**: None (or exact failing `Bi` + file:line)
Files (dazbo-agent-skills)
  • scripts
    • eval_rubric.py 52.2 KB
      Loading…
  • SKILL.md 16.2 KB
    ---
    name: tiered-rubric-loop
    description: |
      Drops a combined deterministic and semantic rubric verification loop into any
      project, acting as the loop component of an agentic harness. Automatically
      leverages existing Spec-Driven Development (SDD) and Test-Driven Development
      (TDD) artefacts - or creates lightweight ones inline - routing each check across
      fast Tier 1 deterministic gates, Tier 1.5 calibrated scoring via the Gemini
      Decisions API (fast and 0 output tokens), and a Tier 2 stateless semantic 
      critique only for 1.5 escalations or complex `[arch]` invariants. 
      Use when running iterative implementation/refinement loops, gating
      PRs or complex slices against `B1`–`Bn` assertions (`SPEC.md` / `LOOPS.md`),
      or scoring diffs and multimodal diagrams.
    metadata:
      author: Darren "Dazbo" Lester
      repository: https://github.com/derailed-dash/dazbo-agent-skills
      cli:
        - uv
    ---
    
    # Tiered Rubric Loop (`tiered-rubric-loop`)
    
    A fast, loosely coupled way to drop a combined **deterministic + semantic verification loop** into any repository as the verification engine of an agentic harness.
    
    It automatically reuses any **Spec-Driven Development (SDD)** (`docs/SPEC.md` or `SPEC.md`) and **Test-Driven Development (TDD)** artefacts already in the repo, or bootstraps a lightweight `B1–Bn` binary rubric checklist inline if none exist.
    
    Instead of waking an expensive reasoning subagent on every single pass, or trusting a zero-thinking classifier blindly on deep security and concurrency invariants, each assertion flows through three escalating stages:
    
    1. **Tier 1 — Deterministic Checks (fast)**: Runs local linters, typecheckers, and unit tests (`make check`, `codespell`, `ruff`, `pyright`, `pytest`) first. Any failure stops the pass immediately so you never waste API calls on broken code.
    2. **Tier 1.5 — Fast Semantic Scoring (fast model, no output tokens)**: Evaluates open binary (`yes`/`no`) rubric criteria (`B1–Bn`) in a single prefill pass using the **Gemini Decisions API** (`gemini-decision-1` via [`scripts/eval_rubric.py`](scripts/eval_rubric.py)). Straightforward editorial and doc-sync checks pass immediately, while obvious omissions fail fast without waking a subagent.
    3. **Tier 2 — Stateless Semantic Critique**: Spawns a fresh, read-only reviewer subagent *only* when Tier 1.5 is uncertain ("grey zone"), or for complex architectural and security invariants (`[arch]`) that need deeper multi-file reasoning.
    
    ## Table of Contents
    
    - [How the 3-Stage Funnel Works](#how-the-3-stage-funnel-works)
    - [Plain-Markdown `B1`–`Bn` Contract & Tagging](#plain-markdown-b1bn-contract--tagging)
    - [Loose Coupling & Companion Skills Matrix](#loose-coupling--companion-skills-matrix)
    - [Bounded Loop Workflow](#bounded-loop-workflow)
    - [Running Tier 1.5 (`eval_rubric.py`)](#running-tier-15-eval_rubricpy)
    - [Threshold Routing & Stateless Tier 2 Critic](#threshold-routing--stateless-tier-2-critic)
    - [Circuit Breaker Trio & Continuous Ratchet](#circuit-breaker-trio--continuous-ratchet)
    - [Loop Receipt Format (`LOOPS.md`)](#loop-receipt-format-loopsmd)
    
    ---
    
    ## How the 3-Stage Funnel Works
    
    | Tier | Mechanism | Latency & Cost | What It Owns |
    | :--- | :--- | :--- | :--- |
    | **Tier 1 (Deterministic)** | Local linters, typecheckers, unit tests (`make check`, `codespell`, `ruff`, `pyright`, `pytest`) | Seconds, **`$0`** | Syntax, static types, formatting, and deterministic test execution. Any failure **short-circuits immediately** before Tier 1.5 or Tier 2 runs. |
    | **Tier 1.5 (Calibrated Probabilistic)** | `gemini-decision-1` via `client.decisions.create` (`v1beta`) in [`scripts/eval_rubric.py`](scripts/eval_rubric.py) | **~150–250 ms**, **`$0.10` / 1M input tokens**, **0 output / thinking tokens** | Evaluates up to 1,000 closed-ended `noul` (yes/no) assertions over a single shared diff/spec/media payload. Terminates clear mechanical passes (`PASS`), rejects obvious omissions immediately (`FAIL_FAST`), and flags uncertain items for L2. |
    | **Tier 2 (Stateless Semantic Critic)** | Fresh read-only `flash` subagent with zero prior coding chain-of-thought | 10–40 s, standard model tokens | Diagnoses grey-zone escalations and performs **one final L2 sign-off** on `[arch]` items after Tier 1.5 is green. |
    
    **Key safety rule**: Because `gemini-decision-1` runs in a single prefill forward pass with **zero thinking tokens** and **zero file-reading tools**, it can produce false positives on subtle logic, race conditions, or auth flaws. Therefore, Tier 1.5 is **terminal** for mechanical/editorial checks, **fail-fast** on clear omissions, and **provisional** for `[arch]` assertions.
    
    ---
    
    ## Plain-Markdown `B1–Bn` Contract & Tagging
    
    The sole shared contract is a Markdown table or bullet list of numbered binary assertions (`B1–Bn`) in `docs/SPEC.md`, `SPEC.md`, or `LOOPS.md` (with optional `--spec <path>` override):
    
    ```markdown
    | ID | Falsifiable Condition | Verification Target |
    | :--- | :--- | :--- |
    | `B1 [det]` | Static gates (`codespell`, `ruff`, `pyright`) and offline tests exit `0` | `make check` |
    | `B2` | `.env.template` and `README.md` document every newly added variable and flag | Doc review |
    | `B3 [arch]` | Auth middleware rejects expired tokens and never retries `4xx` responses | `tests/unit/test_auth.py` |
    ```
    
    Other supported rubric formats and tags:
    
    - **Bullet assertions**: `- **B4**: CLI help panel renders in English (UK)`
    - **Atomic sub-IDs**: `B4a`, `B11b`
    - **Numbered items**: ``1. `B5 [docs]`: ...``
    - **Task checkboxes**: `- [ ] **B6 [concurrency]**: ...` or `- [x] **B6**: ...`
    - **Extended tags**: `[mech]`, `[docs]`, `[ui]`, `[visual]`, `[perf]`, `[concurrency]` (unrecognised tags warn on `stderr` and default safely to `[mech]` instead of silently dropping the assertion)
    
    | Tag | Meaning | Tier 1.5 Handling |
    | :--- | :--- | :--- |
    | `[det]` (or deterministic Verification Target such as `make check`, `pytest tests/...`, `codespell -s`, `ruff check .`) | Already proven by Tier 1 shell exit codes | **Skipped by Tier 1.5** by default (`--include-det` overrides) so the model never guesses whether `pytest` passed from a text diff. |
    | *(untagged, `[mech]`, `[docs]`, `[ui]`, `[visual]`, `[multimodal]`)* | Mechanical, editorial, doc-sync, schema, or UI convention check | **Terminal `PASS`** when `P(yes) >= 0.90`, `confidence >= 0.80`, and **no evidence was truncated**; `FAIL_FAST` when `<= 0.20`; otherwise `ESCALATE_L2_FLASH`. |
    | `[arch]` (also `[security]`, `[logic]`, `[concurrency]`, `[perf]`) | Concurrency, security, state invariant, performance, or test-realism check | **Provisional pass (`PROVISIONAL_ARCH_PASS`)** at Tier 1.5; gated for **one final Tier 2 (`Model: "flash"`) review** once all items clear Tier 1.5. |
    
    Repo-relative file paths and local images named in `Verification Target` or inline backticks are **automatically resolved and attached** (blocking `.env`, secret env files, `*.enc`, and paths outside `--repo-root`; pass `--no-auto-target-files` to disable).
    
    ### Writing High-Signal `B1–Bn` Assertions
    
    - **Keep assertions atomic**: Test **one falsifiable invariant per `Bi`** rather than bundling three functions or eight files into one question—single-invariant assertions give sharp `P(yes)` separation and trigger `FAIL_FAST` accurately on regressions.
    - **Name concrete paths in `Verification Target`**: Prefer `src/auth.py, tests/unit/test_auth.py` over generic `Doc review` / `Code review` so `eval_rubric.py` auto-attaches untouched context files even in narrow working-tree diffs.
    - **Separate canonical registries from summary copy**: Don't assert identical title formatting across both machine registries (`navigation.yaml`) and summarised overview pages (`README.md`, spotlight cards) unless exact parity is truly required.
    
    ---
    
    ## Loose Coupling & Companion Skills Matrix
    
    `tiered-rubric-loop` requires **zero hard dependencies** on other skills and degrades cleanly when optional companions or API keys are absent:
    
    | Companion Skill / Dependency | When Present | Standalone Fallback (When Absent) |
    | :--- | :--- | :--- |
    | `maintaining-core-documentation` / `spec-driven-development` | Reads `B1`–`Bn` directly from `docs/SPEC.md` or `SPEC.md`. | Reads any local `SPEC.md` / `LOOPS.md`, or asks you to agree a compact `B1`–`Bn` list inline before coding. |
    | `test-driven-development` | Delegates the **Act** step to full Red-Green-Refactor execution. | Enforces the core TDD rule inline: write/update a failing test first, change one semantic behaviour per pass, never weaken assertions. |
    | `loopy` | Supplies the outer `Observe -> Choose -> Act -> Verify -> Record -> Stop` governance and `LOOPS.md` receipts. | Runs the identical bounded loop directly with the receipt format below. |
    | `local-code-review` | Stateless Tier 2 read-only subagent runs `local-code-review` on escalated/`[arch]` items. | Uses the built-in **Stateless Tier 2 Critic Prompt** below. |
    | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Enables ~200 ms Tier 1.5 screening via `eval_rubric.py`. | When `eval_rubric.py` exits `3` (missing/invalid key), skips Tier 1.5 cleanly and executes `Tier 1 -> Tier 2` directly while noting how to enable Tier 1.5. |
    
    ---
    
    ## Bounded Loop Workflow
    
    Track every pass around **`Observe -> Choose -> Act -> Verify -> Record -> Stop`**:
    
    1. **Observe**: Read the specification (`docs/SPEC.md`, `SPEC.md`, or `LOOPS.md`), `git status --short`, and current failing tests/checks. Confirm `B1`–`Bn` exist (draft and confirm them first if missing).
    2. **Choose**: Pick **one** highest-value unverified semantic behaviour or failing assertion for this pass (mechanical lint/spelling fixes may be batched).
    3. **Act (TDD)**: Write or update the failing test first (**Red**), confirm it fails, then implement the minimal clean change (**Green**) and refactor without weakening tests.
    4. **Verify (3-Stage Funnel)**:
       - **Tier 1**: Run deterministic checks (`make check` or repo lint + test commands). If Tier 1 fails, **stop verification immediately** and return to **Choose/Act**—do not spend tokens on Tier 1.5 or Tier 2.
       - **Tier 1.5**: Run `eval_rubric.py --json` over the spec, diff, and touched files (plus `--image` / `--pdf` for diagram/UI specs).
       - **Tier 2**: Escalate only grey-zone items immediately, or run a single final `[arch]` critic pass once Tier 1.5 reports zero `FAIL_FAST` and zero grey-zone items.
    5. **Record**: Append pass outcome, `ratchet_score` (mean `P(yes)`), and remaining failures to `LOOPS.md` (or session progress notes).
    6. **Repeat or Stop**: Enforce the **Circuit Breaker Trio** before starting another pass.
    
    ---
    
    ## Running Tier 1.5 (`eval_rubric.py`)
    
    Resolve `scripts/eval_rubric.py` relative to where `tiered-rubric-loop` is installed:
    - Global Antigravity path: `~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py`
    - Project-local path: `.agents/skills/tiered-rubric-loop/scripts/eval_rubric.py`
    
    In sandboxed agent runtimes that block outbound HTTPS by default, invoke `uv run` with unsandboxed network permission (`BypassSandbox: true` in Antigravity) for live API calls, or pass `--offline` for sandboxed `--dry-run` checks.
    
    ```bash
    # 1. Working-tree loop: auto-discover SPEC.md / docs/SPEC.md / LOOPS.md
    uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --json
    
    # 2. Branch or merged PR gating: evaluate committed changes against base ref / range
    uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --base main --json
    uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
      --git-range origin/main...HEAD \
      --only B3,B4 \
      --json
    
    # 3. Explicit spec, scoped files, and multimodal diagram/UI screenshot check
    uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
      --spec docs/SPEC.md \
      --file src/auth.py \
      --image docs/images/architecture.png \
      --json
    
    # 4. Offline dry-run inside sandbox (verifies rubric, targets, and truncation zero-API)
    uv run --offline ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --dry-run --json
    ```
    
    For large PRs (`> 8` files or `> 100 KB`), shard evaluations by assertion group via `--only B3,B4` so files stay within `--max-file-chars` (`24,000`) and `--max-evidence-chars` (`180,000`). If any file or combined payload is truncated, `eval_rubric.py` populates `truncated_sources` and **downgrades mechanical `PASS` to `ESCALATE_L2_FLASH`** unless `--allow-truncated-pass` is explicitly set.
    
    ### Exit Codes
    
    | Code | Meaning | Loop Action |
    | :--- | :--- | :--- |
    | `0` | Tier 1.5 clear (all evaluated items are `PASS` or `PROVISIONAL_ARCH_PASS`) | If `final_l2_arch_ids` is non-empty, run **one** Tier 2 (`Model: "flash"`) critic pass on those IDs; otherwise exit loop as **`Success`**. |
    | `1` | One or more `FAIL_FAST` or `ESCALATE_L2_*` items (or API call error) | Address `FAIL_FAST` items directly; escalate grey-zone items (`ESCALATE_L2_*`) only when root cause is non-obvious. |
    | `2` | Usage error, missing spec, or no `B1`–`Bn` assertions parsed | Fix spec path or rubric formatting before continuing. |
    | `3` | Missing/invalid `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Degrade gracefully to **`Tier 1 -> Tier 2`** and inform the user once. |
    
    ---
    
    ## Threshold Routing & Stateless Tier 2 Critic
    
    Default calibrated thresholds (overridable via `--pass-threshold`, `--fail-threshold`, `--min-confidence`):
    - **`PASS`** (`P(yes) >= 0.90`, `confidence >= 0.80`, mechanical): **Terminal pass.** No L2 model runs.
    - **`PROVISIONAL_ARCH_PASS`** (`P(yes) >= 0.90`, `confidence >= 0.80`, `[arch]`): Held until all other assertions pass, then verified once by a Tier 2 (`Model: "flash"`) critic.
    - **`FAIL_FAST`** (`P(yes) <= 0.20`, `confidence >= 0.80`): Returned directly to the coding agent—do **not** burn an L2 turn repeating an obvious omission.
    - **`ESCALATE_L2_FLASH`** (`0.20 < P(yes) < 0.90` or `confidence < 0.80`): Escalate **only those specific `Bi` IDs** to a fresh read-only subagent.
    
    ### Subagent Routing Across Coding Agents
    
    Always spawn Tier 2 as a **fresh, stateless, read-only subagent** that never sees the coding agent's prior chain-of-thought or failed attempts. When spawning, use a fast or flash agent. For example, in **Antigravity**, call `invoke_subagent` with `TypeName: "research"` and `Model: "flash"`.
    
    ### Built-In Stateless Tier 2 Critic Prompt
    
    Pass this exact prompt (filling in only the target `Bi` IDs, spec path, and changed files):
    
    ```text
    You are a stateless, read-only Tier 2 semantic critic. You have NOT seen the
    coding session and must judge solely from the specification and actual files on disk.
    
    Evaluate ONLY these rubric assertions: [INSERT B_i IDs AND CONDITIONS].
    Read: [SPEC.md / docs/SPEC.md] and inspect the changed/untracked files + git diff.
    
    Rules:
    1. Cite exact file paths, line numbers, and observed code/test behaviour.
    2. Check whether tests verify real behaviour rather than mocking away logic.
    3. For each assertion ID, return PASS or FAIL with a concise root-cause note.
    4. Do not invent fixes or assume uninspected files pass.
    ```
    
    ---
    
    ## Circuit Breaker Trio & Continuous Ratchet
    
    Stop the loop whenever **any** of these three conditions triggers:
    
    1. **100% Pass**: Tier 1 passes, Tier 1.5 reports `exit_code == 0`, and any `final_l2_arch_ids` clear their final Tier 2 (`Model: "flash"`) review (**`Success`**).
    2. **Pass Budget Exhausted**: Maximum **4–5 passes** per slice reached (**`Exhausted`**—escalate remaining blockers to the user).
    3. **2 Consecutive Non-Improving Passes**: Neither failing assertion count decreases **nor** `ratchet_score` (mean `P(yes)` across evaluated `B1`–`Bn` items) improves across two consecutive passes (**`No progress`**—escalate to the user rather than spinning).
    
    Tracking the continuous `ratchet_score` avoids premature stalls when a pass genuinely moves an assertion from e.g. `P(yes) = 0.25` to `0.82` while still short of `0.90`, and catches silent regressions immediately.
    
    ---
    
    ## Loop Receipt Format (`LOOPS.md`)
    
    Record completed runs (or append per-pass telemetry) concisely:
    
    ```markdown
    ## Tiered Rubric Loop Receipt
    
    - **Target**: [Slice / Feature / PR]
    - **Spec**: [docs/SPEC.md] (`B1`–`Bn`)
    - **Passes**: [N / 5] | **Outcome**: `Success` | `Exhausted` | `No progress` | `Blocked`
    - **Tier 1**: `make check` -> PASS
    - **Tier 1.5 Ratchet (`gemini-decision-1`)**: `0.58 -> 0.94 -> 0.98` (skipped `[det]`: `B1`)
    - **Tier 2 (`Model: "flash"`)**: Verified `[arch]` IDs `B3` -> PASS
    - **Remaining / Escalation**: None (or exact failing `Bi` + file:line)
    ```
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.