{"slug":"codex-ab","title":"codex-ab","summary":"Run an A/B codex review experiment — holistic codex review vs 3 focused dimension passes (security, ecto, liveview) on the branch diff, classify findings, report a panel-value verdict. Use when the branch is fresh, before any codex review runs.","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-10-04T15:14:19.306197Z","repo":{"url":"https://github.com/oliver-kriska/claude-elixir-phoenix","stars":560,"forks":44,"license":"MIT","updatedAt":"2026-10-02T04:11:38Z"},"bodyHtml":"
\n

name: codex-ab\ndescription: Run an A/B codex review experiment — holistic codex review vs 3 focused dimension passes (security, ecto, liveview) on the branch diff, classify findings, report a panel-value verdict. Use when the branch is fresh, before any codex review runs.\neffort: medium\nargument-hint: \"[base-branch]\"

\n

Codex Panel A/B (contributor instrument — verdict decided)

\n

Answer one question with evidence: do dimension-focused codex passes find\nreal issues that one holistic codex exec review misses? Runs both on the\nsame diff, then classifies every focused finding against the holistic pass.

\n
\n

DECIDED 2026-07-10 after 4 runs (2 fresh): panel KILLED. Fresh-only\n1 real miss / 1 false positive plus one zero-value run at 4× cost —\nreal misses did not outnumber FPs. Kept as contributor tooling (NOT\ndistributed) for one possible retest: a UI-heavy diff with a single\nextra liveview-focused pass (2× cost). Scoreboard:\n.claude/research/2026-07-03-codex-review-integration.md §7.

\n
\n

Usage

\n
/codex-ab            # A/B against main (~5 min, 4 codex runs)\n/codex-ab develop    # explicit base branch\n
\n

Iron Laws

\n
    \n
  1. FRESH DIFF ONLY — ask the user to confirm this branch has NOT been\ncodex-reviewed yet (cloud or /phx:codex-loop). A drained diff returns\nNO FINDINGS everywhere and proves nothing — wasted quota
    2. \n
  2. Verify every REAL MISS in the code before counting it — a focused\nfinding only scores if the issue actually exists at that file:line
    3. \n
  3. Read ONLY the findings .md files — streams are diverted to .log\nfiles; never cat a log into context (10k+ lines each)
    4. \n
  4. Exactly 4 codex runs, never re-run dimensions — bounded quota
    5. \n
  5. Persist the verdict — an unrecorded experiment is wasted quota
    6. \n
\n

Workflow

\n

Step 1: Preflight

\n

Run command -v codex — missing → STOP with install hint. Then:

\n\n

Step 2: Run the A/B (background, ~5 min)

\n
bash ${CLAUDE_SKILL_DIR}/scripts/codex-panel-ab.sh {base} \\\n  .claude/reviews/codex-ab-$(date +%Y-%m-%d-%H%M)\n
\n

Use run_in_background — it runs 1 holistic codex exec review + 3\nfocused codex exec workers (security / ecto / liveview) in parallel,\nall streams redirected. Do other work or wait; never poll.

\n

Step 3: Classify

\n

Read the 4 findings files (holistic.md, security.md, ecto.md,\nliveview.md — small). For EACH focused finding:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ClassMeaningTest
DUPLICATEHolistic already found itSame file + same defect
REAL MISSGenuine issue holistic missedRead the code at file:line — defect confirmed (Iron Law 2)
FALSE POSITIVEManufactured, pre-existing, or wrongCode check fails, or issue exists on base branch too
\n

Step 4: Verdict

\n

Present:

\n
## Codex Panel A/B — {branch} vs {base}\n| dimension | findings | duplicate | real miss | false positive |\nHolistic-only findings: {n}\nVerdict this run: {REAL MISS count} real miss vs {FP count} false positive\nDecision rule: build --codex-panel only if real misses outnumber false\npositives across 2-3 fresh branches.\n
\n

Write the verdict table to .claude/reviews/codex-ab-{date}/VERDICT.md.\nSuggest repeating on the next 1–2 fresh branches before deciding.

\n

Integration

\n
fresh branch → /codex-ab (YOU ARE HERE) → verdict logged\n   ├─ real misses win across runs → build /phx:review --codex-panel\n   └─ duplicates/FPs win → keep holistic /phx:codex-loop, drop panel idea\n       └─ OUTCOME 2026-07-10: this branch won — panel dropped\n
\n

References

\n\n","files":[{"path":"scripts/codex-panel-ab.sh","sizeBytes":3330,"isText":true},{"path":"SKILL.md","sizeBytes":3970,"isText":true},{"path":"triggers.json","sizeBytes":719,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-10-04T15:15:46.51213Z","sha256":"546013A2A3B10CE72099EB05C90A2F409E1F44A735C9CCA88E1CA151D4D4299A","sizeBytes":4361},"review":null,"source":{"repositoryUrl":"https://github.com/oliver-kriska/claude-elixir-phoenix","path":".claude/skills/codex-ab","license":"MIT","commit":"9767a82d24ddddad553e85f88efc2869a7fd7d88","subtreeSha":"F0C5C9BF812FD165473F931D909ACDF38C12497F8509280016C8ABE2F268B9EB","lastSyncedAt":"2026-10-04T15:14:09.139242Z"},"reviewedAt":"2026-10-04T15:18:17.909331Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/oliver-kriska/claude-elixir-phoenix/tree/main/.claude/skills/codex-ab"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oliver-kriska-claude-elixir-phoenix@llmmart"},{"target":"git","command":"git clone https://github.com/oliver-kriska/claude-elixir-phoenix.git"}]}