# Iteration 1 -- Compound Technical+Consumer (Empirical)

## Plan

**Type:** Empirical test (compound question)

**Question:** "What are the leading open-source LLM inference engines, and which is best
suited for a small team deploying a 70B parameter model on consumer GPUs?"

**What this tests:**

- Phase 0: Question type classification (should detect Technical + Consumer compound)
- Phase 1: Compound decomposition and presentation to user
- Phase 2: Scope calibration (compound should escalate to Broad)
- Phase 3: Team setup, output directory creation, task creation
- Phase 4-6: Multi-round investigation with follow-up
- Phase 8: Mixed synthesis (Technical comparison + Consumer recommendation)
- Phase 9: File persistence (state.md, researcher findings, report.md)

**Rationale:** First-ever empirical run of the v2 skill. A compound question exercises the
most distinctive new features. The topic is genuinely useful and has enough complexity to
stress-test the system without being so niche that sources are sparse.

## Research Summary

Ran Broad-scope research (3 researchers, 2 rounds) on "What are the leading open-source LLM
inference engines, and which is best suited for a small team deploying a 70B parameter model
on consumer GPUs?" Full report at `~/.claude/research/llm-inference-engines-70b-consumer-gpu/report.md`.

**Round 1:** 3 researchers investigated landscape (A), benchmarks (B), practical deployment (C).
Strong convergence on vLLM for multi-GPU, llama.cpp/Ollama for single-GPU. Good source diversity
(25+ sources across researchers).

**Round 2:** 2 targeted follow-ups -- SGLang vs vLLM comparison (B) and ExLlamaV3 viability (A).
Resolved conflicting throughput claims and assessed ExLlamaV3 maturity.

**Outcome:** Comprehensive report with Technical comparison matrix + Consumer ranked recommendations.
25 cited sources. Report quality is genuinely useful.

## Observations

**What worked well:**

1. **Phase 0 classification worked correctly.** Identified compound (Technical + Consumer), chose
   Broad scope. No friction.

2. **Researcher independence produced good diversity.** Each researcher found unique sources with
   minimal overlap. Cross-referencing in triage revealed genuine convergence (not echo chamber).

3. **Round 2 follow-ups were well-targeted.** The SGLang vs vLLM task resolved a real conflict
   from Round 1 (3.1x claim traced to non-independent benchmarks). ExLlamaV3 assessment filled
   a genuine gap.

4. **Source quality was generally high.** Researchers followed type-aware source evaluation --
   prioritized official docs, controlled benchmarks, and practitioner reports over marketing.

5. **File persistence worked.** state.md, researcher findings, and report.md all written correctly.

**What was clunky or problematic:**

1. **Researchers didn't self-claim follow-up tasks.** The skill says researchers should check
   TaskList after completing their initial task and self-claim available work. In practice, all
   3 researchers went idle after Round 1 and had to be explicitly nudged with SendMessage to
   pick up Round 2 tasks. Researcher B went idle even after being sent a direct message about
   Task #7 and needed a second nudge. **This is the biggest friction point.**

2. **Compound decomposition wasn't presented in the exact template format.** The skill says to
   present "> This breaks down into {N} sub-questions: ..." but I presented it slightly
   differently. Minor -- the user understood fine.

3. **No researcher findings files from Round 1.** The skill says researchers should write to
   `{output_dir}/researcher-{letter}-findings.md`. Only Researcher B wrote a findings file for
   Round 2 (SGLang). Researchers A, B, C did NOT persist their Round 1 findings to disk. The
   findings were only delivered via SendMessage. **This breaks the resume capability** -- if the
   session crashed after Round 1, only state.md would exist, not the underlying findings.

4. **Researcher C was underutilized in Round 2.** With only 2 follow-up tasks and 3 researchers,
   Researcher C sat idle. The skill doesn't give clear guidance on whether to shut down unused
   researchers early or find work for them.

5. **Team task list vs iteration loop task list collision.** The research team uses TaskCreate
   for research angles, but the iteration loop also uses TaskCreate for tracking iteration
   progress. These two task lists coexisted in the same session (team tasks #1-8, iteration
   tasks #1-5 in a different context). This worked but could be confusing in a resumed session.

6. **The skill doesn't specify the lead's name for researchers to message.** Researchers need
   to read the team config to find the lead's name. The researcher-prompt.md says "The lead's
   name is typically the team creator. Check the team config if unsure." This is indirect --
   could be explicit in the spawn prompt.

7. **Synthesis was a single monolithic write.** At ~250 lines, the report was written in one
   Write call. For longer reports, this could fail or be hard to review. The skill doesn't
   address progressive synthesis or draft-then-refine.

**Surprising or noteworthy:**

- Researchers found genuinely useful, current sources (Jan-Feb 2026 content). Exa semantic
  search worked well for this technical topic.
- The compound question naturally produced a well-structured report (Part 1: Technical
  Comparison, Part 2: Consumer Recommendation). The type-aware templates mapped cleanly.
- Total research time was reasonable for a Broad scope run (~10-15 minutes wall clock).

## Changes Made

**1. Restructured researcher reporting flow** (`references/researcher-prompt.md`)

- Replaced the old "Reporting Findings" + "Task Management" sections with a clear 3-step
  sequence: (1) persist to disk, (2) SendMessage to lead, (3) mark complete + TaskList.
- Moved file persistence from buried rule #9 to Step 1 of the main reporting flow.
- Added explicit filenames for follow-up tasks: `researcher-{letter}-findings-{task-id}.md`.

**2. Added emphatic task self-claim section** (`references/researcher-prompt.md`)

- New "Task Self-Claim (IMPORTANT)" section with explicit instructions to always check
  TaskList after completing each task and immediately claim available work.
- Added "Do NOT go idle if there are available tasks" instruction.

**3. Updated Critical Rules** (`references/researcher-prompt.md`)

- Replaced rules #6-9 with two focused rules: "Persist, THEN report, THEN check for work"
  and "ALWAYS self-claim available tasks". Removed redundant rule about SendMessage (now
  covered in main flow).

**4. Added lead name to spawn prompt template** (`SKILL.md`)

- Added `Lead name: team-lead` line to the researcher spawn prompt so researchers don't
  need to look up the team config to know who to message.
- Added `IMPORTANT: After completing each task, ALWAYS check TaskList...` reminder at the
  end of the spawn prompt for reinforcement.

**5. Added settings.json permission** (`configs/claude/settings.json`)

- Added `Bash(mkdir -p ~/.claude/research/*)` to avoid permission prompts during research
  output directory creation.

## Notes for Next Iteration

- The self-claim fix is the most important change but also the hardest to verify -- it depends
  on researcher agent behavior, which may not fully follow instructions regardless of emphasis.
  Iteration 2 should test whether the changes actually reduce the nudging problem.
- Compound synthesis quality was good -- the two-part structure (Technical + Consumer) mapped
  naturally. No changes needed to synthesis templates yet.
- The Contested question test (backlog #2) would exercise completely different skill features
  (steelmanning, cross-agent verification, Comprehensive scope). Good candidate for Iteration 2.
- New backlog idea: investigate whether spawn prompt length affects researcher behavior.
  The prompt is already ~15 lines; adding more "IMPORTANT" reminders has diminishing returns.

# Iteration 2 -- Contested Question (Empirical)

## Plan

**Type:** Empirical test (Contested question)

**Question:** "Should AI model weights be open-sourced?"

**What this tests:**

- Phase 0: Contested classification (should trigger Comprehensive scope)
- Phase 1: Research plan presentation (Contested template with angles)
- Phase 2: Comprehensive calibration (4 researchers, 2-3 rounds, opus model)
- Phase 7: Cross-agent verification (unique to Comprehensive)
- Phase 8: Contested synthesis template (steelmanning, no verdict)
- **Iteration 1 fixes:** researcher self-claim, file persistence, lead name in spawn prompt

**Also validates:**

- Context compaction resilience (we're at ~60% context, likely to compact mid-run)
- Whether emphasizing self-claim in researcher-prompt.md reduces nudging

## Research Summary

Ran Comprehensive-scope research (4 researchers, 3 rounds) on "Should AI model weights be
open-sourced?" Full report at `~/.claude/research/ai-model-weights-open-source/report.md`.

**Round 1:** 4 opus researchers investigated: (A) pro-openness arguments, (B) anti-openness
arguments, (C) regulatory/governance landscape, (D) empirical evidence. Strong convergence on
key structural points (irreversibility, safeguard removability, safety research enablement).
60+ sources across researchers.

**Round 2:** 2 targeted follow-ups -- graduated release proposals (A) and RAND CBRN contradiction
(D). Resolved the RAND 2024 vs 2025 discrepancy (different models, different methodologies,
both partially right). Found rich graduated release literature (Solaiman gradient, structured
access, Carnegie consensus, MOF, RSPs).

**Round 3 (Verification):** 2 cross-agent verifications --

- SentinelOne 7.5% claim (verified by B): SUPPORTED WITH NUANCE. Lower bound on a subset,
  corroborated by 2 independent studies (Censys solo, Cisco Talos).
- Greenblatt ~100K fatalities estimate (verified by C): CONTESTED. Independent models produce
  ~10K-60K range, but direction agreed and uncertainty intervals overlap.

**Outcome:** 37-footnote Contested synthesis report with steelmanned positions, points of
agreement, key disagreements (factual and values-based), evidence quality assessment. No verdict
rendered. Report quality is genuinely useful for someone trying to understand the debate.

## Observations

**What worked well:**

1. **Phase 0 classification worked correctly.** Identified Contested type, selected Comprehensive
   scope (4 researchers, opus model, 2-3 rounds + verification). No friction.

2. **File persistence fix from Iteration 1 worked perfectly.** All 4 researchers wrote findings
   to disk in Round 1: `researcher-a-findings.md`, `researcher-b-findings.md`,
   `researcher-C-findings.md`, `researcher-D-findings.md`. Round 2 follow-ups also persisted:
   `researcher-a-findings-9.md`, `researcher-D-findings-10.md`. Verifications persisted:
   `researcher-b-findings-12.md`, `researcher-C-findings-11.md`. **This is a clear win from
   Iteration 1 changes.**

3. **Cross-agent verification added genuine value.** The Greenblatt CBRN verification (CONTESTED)
   surfaced 4 independent quantitative models (GovAI/Righetti, GovAI/Williams, Jones, RAND Delphi)
   that put the estimate in context. The SentinelOne verification (SUPPORTED WITH NUANCE) traced
   the claim to its primary source and found Reuters had oversimplified. Both verdicts materially
   improved the final report.

4. **Researcher quality was excellent at opus level.** Source diversity, depth of analysis, and
   critical evaluation were noticeably stronger than the sonnet researchers in Iteration 1.
   Researchers proactively labeled institutional biases, distinguished methodology types, and
   cross-referenced within their own findings.

5. **Contested synthesis template worked well.** The steelmanned positions format naturally
   organized the findings. Separating factual disagreements from values disagreements was
   particularly effective. The "Evidence Quality by Position" section forced honest assessment
   of both sides' weaknesses.

6. **Lead name in spawn prompt worked.** Researchers messaged "team-lead" directly without
   needing to look up the team config. No friction observed.

7. **Context compaction survived.** The session compacted mid-run (between Round 1 triage and
   Round 2 dispatch). The ITERATION-LOG.md and state.md files were sufficient to resume
   coherently. The compacted context summary preserved key details about researcher findings
   and verification assignments.

**What was clunky or problematic:**

1. **Self-claim fix still did not work reliably.** Despite the emphatic "Task Self-Claim
   (IMPORTANT)" section and the spawn prompt reminder, researchers still went idle after
   completing their Round 1 tasks without checking TaskList. The lead had to explicitly create
   follow-up tasks and (in the pre-compaction portion) likely nudge researchers to claim them.
   After compaction, the lead dispatched Round 2 and Round 3 tasks by sending direct messages
   to specific researchers rather than relying on self-claim. **This is now a confirmed pattern
   across 2 iterations: researchers do not reliably self-claim tasks regardless of how
   emphatically the instructions say to do so.**

2. **Inconsistent filename casing.** Researchers wrote `researcher-a-findings.md` (lowercase)
   and `researcher-C-findings.md` (uppercase). The researcher-prompt.md says
   `researcher-{letter}-findings.md` without specifying case. The spawn prompt uses uppercase
   letters (A, B, C, D) while the template uses lowercase `{letter}`. This inconsistency is
   cosmetic but could cause issues for scripts or resume logic that expect consistent naming.

3. **Researcher C required double shutdown request.** The first shutdown_request was sent but
   C didn't respond until a second was sent. This may be a timing issue (C was still processing)
   rather than a systematic problem. Researchers A, B, D all shut down on first request.

4. **Round 2 researcher allocation was suboptimal.** With 2 follow-up tasks and 4 researchers,
   B and C sat idle during Round 2. Then in Round 3, B and C were assigned verification tasks.
   The skill doesn't provide guidance on whether to assign Round 2 tasks to the SAME researchers
   who covered related Round 1 angles (continuity) or DIFFERENT researchers (fresh perspective).
   For follow-ups, continuity seems better; for verification, fresh perspective is required.

5. **No guidance on when to stop adding rounds.** The skill says "2-3 rounds" for Comprehensive
   but doesn't specify criteria for choosing 2 vs 3. In practice, verification (Round 3) added
   clear value for the two high-impact single-source claims. But the decision was ad hoc.

**Surprising or noteworthy:**

- Opus researchers found genuinely high-quality, current sources including January-February 2026
  content. The contested topic prompted researchers to find primary sources rather than relying
  on secondary reporting.
- The RAND CBRN contradiction investigation was the most valuable follow-up: it found a THIRD
  RAND study (Delphi panel) and a FOURTH (benchmarking study) that helped reconcile the apparent
  contradiction. This would have been missed without targeted follow-up.
- Report length (~350 lines, 37 footnotes) is longer than ideal. The Contested template naturally
  produces longer output because steelmanning both sides requires more space. The skill doesn't
  address length targets for different question types.

## Changes Made

**1. Replaced self-claim with lead-dispatched follow-ups** (`SKILL.md` Phase 6, `researcher-prompt.md`)

- The biggest architectural change. After 2 iterations confirming researchers don't self-claim
  regardless of instruction emphasis, switched to a model where the lead creates tasks AND
  explicitly messages each researcher to assign them.
- SKILL.md Phase 6: Added "Assignment strategy" section with guidance on who gets follow-ups
  (continuity for deep-dives, fresh perspective for conflicts).
- SKILL.md Phase 7: Added same lead-dispatched pattern for verification tasks.
- researcher-prompt.md: Replaced emphatic "Task Self-Claim (IMPORTANT)" section with brief
  "Between Tasks" section. Now says: check TaskList once, claim if available, otherwise go
  idle and wait for lead to message you.
- researcher-prompt.md: Updated Critical Rules to remove self-claim rule.

**2. Normalized filename casing to lowercase** (`SKILL.md`, `researcher-prompt.md`)

- Spawn prompt now says `(use LOWERCASE in filenames: researcher-{lowercase letter})`
- researcher-prompt.md Step 1: Added "(always use **lowercase** for the letter)"
- researcher-prompt.md Critical Rules: Added rule #7 "Use lowercase filenames"

**3. Added round stopping criteria** (`SKILL.md` Phase 7)

- Added explicit "When to run Round 3" and "When to skip Round 3" criteria to Phase 7.
- Run verification when: high-impact single-source claims, unresolved factual conflicts,
  or specialized/niche topic claims.
- Skip verification when: no single-source high-impact claims remain, conflicts resolved by
  Round 2, remaining uncertainties are about magnitude not direction.

**4. Added Round 2 assignment guidance** (`SKILL.md` Phase 6)

- Deep-dives/gap-fills: assign to researcher with related Round 1 angle (continuity)
- Conflict resolution: assign to researcher who did NOT cover either side (fresh perspective)
- Idle researchers: wait for verification or shut down early

## Notes for Next Iteration

- The lead-dispatched model is the most important change to validate in Iteration 3. It
  accepts the reality that researchers don't self-claim and makes the lead's role explicit.
  This should eliminate the "nudging" friction entirely.
- Iteration 2 demonstrated that Comprehensive scope with opus researchers produces excellent
  quality. The main question is whether the cost (4 opus researchers x 3 rounds) is justified
  for less complex questions.
- The Contested synthesis template is naturally verbose (~350 lines, 37 footnotes). Consider
  adding length targets per question type in a future iteration.
- Good candidate for Iteration 3: Emerging/Frontier test (backlog #3) would exercise
  instability warnings, date-sensitive source evaluation, and the Emerging synthesis template.
  It would also test the lead-dispatched model with a different question type.
- Backlog item #8 (spawn prompt length) can be removed -- the self-claim problem was resolved
  by changing the architecture rather than tweaking prompt emphasis.

# Iteration 3 -- Consumer Comparison (Empirical)

## Plan

**Type:** Empirical test (Consumer question, non-AI domain)

**Question:** "Best mechanical keyboard for programming in 2026?"

**What this tests:**

- Phase 0: Consumer classification (should trigger Broad scope by default)
- Phase 1: Brief clarification (Consumer type, simple question)
- Phase 2: Broad calibration (3 researchers, 2 rounds, sonnet model)
- Phase 8: Consumer synthesis template (ranked recommendations)
- **Iteration 2 fix:** lead-dispatched follow-ups (the biggest change to validate)
- **Domain shift:** non-AI topic tests whether Exa search and researcher methodology work
  outside our comfort zone

**Also validates:**

- Expert testing lab prioritization in source evaluation (Consumer type hierarchy)
- Whether sonnet researchers (cheaper) produce adequate quality for a simpler topic
- Whether Broad scope is sufficient for a Consumer question (or if it needs less)

## Research Summary

Ran Broad-scope research (3 sonnet researchers, 2 rounds) on "Best mechanical keyboard for
programming in 2026?" Full report at `~/.claude/research/best-mech-keyboard-programming-2026/report.md`.

**Round 1:** 3 researchers investigated: (A) expert reviews/testing, (B) community
recommendations, (C) features/specs analysis. Strong convergence on Keychron dominance at
every price tier. 30+ sources across researchers.

**Round 2:** 2 targeted follow-ups using lead-dispatched model:

- Split/ergonomic deep-dive (C, continuity): Consolidated Voyager vs Glove80 vs Defy vs Go60.
  Discovered MoErgo Go60 as significant new entrant. Alice layout as middle ground.
- Blind spot check (A, fresh perspective): Mac-specific picks (NuPhy Air75 V2 top for Mac),
  office noise reality (no mechanical keyboard is truly quiet), Leopold as overlooked quality.

**Outcome:** Consumer-template report with ranked recommendations (Top Pick: Q5 Max, Runner-Up:
V5 Max, Budget: C3 Pro, Best Mac: NuPhy Air75 V2, Best Ergonomic: Glove80, Best Gentle
Upgrade: Keychron Q8 Max Alice, Best Quiet: HHKB Type-S). 25 cited sources. Report is
genuinely useful as a buying guide.

## Observations

**What worked well:**

1. **Lead-dispatched follow-ups worked perfectly on the first try.** This is the most important
   validation from this iteration. The lead created 2 Round 2 tasks, then sent direct
   SendMessage to researcher-c (split deep-dive, continuity) and researcher-a (blind spots,
   fresh perspective). Both researchers claimed their tasks and began immediately with zero
   nudging. Researcher-b was correctly left idle (2 tasks < 3 researchers). **The Iteration 2
   architectural change eliminated the self-claim problem completely.**

2. **Lowercase filename normalization worked.** All 5 findings files used correct lowercase
   naming: `researcher-a-findings.md`, `researcher-b-findings.md`, `researcher-c-findings.md`
   (Round 1), `researcher-c-findings-7.md`, `researcher-a-findings-8.md` (Round 2). The
   Iteration 2 fix (explicit lowercase instructions in spawn prompt + researcher-prompt.md)
   resolved the inconsistency.

3. **Consumer synthesis template produced a well-structured report.** The ranked recommendations
   format (Top Pick, Runner-Up, Budget, Best for X) naturally organized the findings into
   actionable buying advice. Each pick has evidence from multiple sources and clear reasoning.

4. **Non-AI domain worked fine.** Exa semantic search handled keyboard/hardware topics well.
   Researchers found high-quality sources including RTINGS lab testing (278 keyboards), Wirecutter,
   Reviewed.com, WIRED, plus community sources (Reddit 381K opinion analysis, HN, Devtalk).
   No domain-specific search issues.

5. **Sonnet researcher quality was adequate for Consumer scope.** Findings were well-structured,
   sources properly evaluated, and cross-source analysis was competent. The quality difference
   from opus (Iteration 2) was noticeable but acceptable: sonnet researchers were less likely
   to proactively label biases or trace claims to primary sources, but the overall research
   was solid.

6. **All 3 researchers shut down cleanly on first request.** No double-shutdown needed (unlike
   Iteration 2 where researcher-c required two attempts). This may have been a timing
   improvement or just variance.

7. **Assignment guidance worked as designed.** Split deep-dive went to researcher-c (who covered
   splits in Round 1 -- continuity). Blind spot check went to researcher-a (who covered expert
   reviews -- fresh perspective on what was missed). The guidance in Phase 6 made assignment
   decisions easy and justified.

**What was clunky or problematic:**

1. **Round 1 had no gaps requiring urgent follow-up.** The Broad scope consumer question
   produced such strong convergence in Round 1 (Keychron dominance was overwhelming) that
   Round 2 follow-ups were useful but not essential. The split/ergonomic deep-dive and blind
   spot check added genuine value (Go60 discovery, Mac-specific picks, noise reality check),
   but the core recommendations would have been the same without them. **This suggests Focused
   scope (1 round) might be sufficient for straightforward Consumer questions.**

2. **Researcher A idle-looped before claiming task.** After receiving the follow-up assignment
   via SendMessage, Researcher A sent 2 idle notifications before starting work. The task was
   eventually claimed and completed successfully, so this is cosmetic, but the idle chatter is
   noisy in the notification stream.

3. **Report length was ~200 lines, 25 footnotes.** More manageable than the Contested report
   (~350 lines, 37 footnotes) but still substantial. The Consumer template's multiple "Best
   for X" categories naturally expand the report. No length target exists in the skill.

4. **Phase 1 clarification was completely skipped.** The skill says "Brief clarification only
   if genuinely ambiguous" for Consumer questions. The question was clear enough to skip
   entirely, which is correct behavior, but it means the user had no opportunity to specify
   constraints (budget range, layout preference, Mac vs Windows, etc.) that could have focused
   the research. For Consumer questions, a quick "Any constraints?" might be valuable even
   when the question seems clear.

**Surprising or noteworthy:**

- The dharm.is analysis of 381K Reddit opinions (found by researcher-b) was an unexpectedly
  rich quantitative source for a Consumer topic. It revealed the enthusiast vs practical
  programmer preference gap that informed the report's framing.
- The MoErgo Go60 (found by researcher-c in Round 2) was genuinely new information -- launched
  late 2025, only one detailed review exists. This is exactly the kind of find that makes
  Round 2 follow-ups worthwhile even when Round 1 converges.
- Total wall-clock time was ~15 minutes for the full Broad scope run, similar to Iteration 1.

## Changes Made

**No skill file changes this iteration.** The Iteration 2 changes (lead-dispatched follow-ups,
lowercase filenames, round stopping criteria, assignment guidance) all validated successfully.
The observations suggest potential improvements but none urgent enough to change now:

- Scope calibration for simple Consumer questions (Focused might suffice) -- worth investigating
  as a meta-research topic rather than a code change
- Consumer-specific Phase 1 constraint gathering -- minor UX improvement, not blocking
- Idle notification noise -- cosmetic, not actionable in the skill

## Notes for Next Iteration

- **Lead-dispatched model is confirmed working across 2 question types.** The self-claim
  problem is solved. No further changes needed to this architecture.
- **All Iteration 2 changes validated.** Lowercase filenames, assignment guidance, and round
  stopping criteria all worked as designed. The skill is now stable for these features.
- **Good candidate for Iteration 4:** Meta-research on scope calibration. Three iterations
  have now produced data on scope appropriateness: Iteration 1 (Broad for compound Technical +
  Consumer, felt right), Iteration 2 (Comprehensive for Contested, felt right but expensive),
  Iteration 3 (Broad for Consumer, Round 2 was useful but not essential). Could the skill
  provide better guidance on when Focused is sufficient vs when Broad is needed?
- **Alternative candidate:** Emerging/Frontier test (backlog #3) to exercise the last untested
  synthesis template. Or information cascade detection (backlog #4) as the first meta-research
  iteration.
- **The skill is approaching diminishing returns for empirical testing.** Three successful runs
  across 3 question types (compound Technical+Consumer, Contested, Consumer) with improving
  results each iteration. The remaining untested templates (Scientific/Health, Emerging/Frontier,
  Opinion/Sentiment, Factual) follow the same architecture. Meta-research may yield higher
  marginal value than additional empirical runs.

# Iteration 4 -- Search Tool Audit + Access Workarounds (Meta-Research)

## Plan

**Type:** Meta-research (run through the team skill, not manual)
**Backlog item:** #4
**Scope:** Broad (3 sonnet researchers, 2 rounds)

**Goal:** Audit the full Exa and Firecrawl tool sets for underused features, research
academic search options and paywall workarounds, then update `researcher-prompt.md` with
improved guidance.

## Research Summary

**Report:** `~/.claude/research/web-research-tool-effectiveness/report.md`

Round 1 angles: (A) Exa/Firecrawl advanced features, (B) content access workarounds,
(C) academic search strategies. Round 2 follow-ups: (A) optimal Exa+Firecrawl workflow
decision tree, (B) blind spot check (OpenAlex, Firecrawl limitations, additional APIs).
Researcher C shut down early (no follow-up needed). 24+ sources, 5 findings per researcher.

**Critical finding:** Both Exa and Firecrawl MCP tools expose only a subset of their full
API capabilities. **Resolved post-iteration:** Switched Exa from npm package to hosted HTTP
endpoint (`mcp.exa.ai/mcp`) with `web_search_advanced_exa` and `people_search_exa` enabled.
This exposed all missing parameters (category, includeDomains/excludeDomains, date filtering,
highlights/summary). Firecrawl search was then removed from the researcher prompt as
redundant -- Exa advanced covers domain targeting, date filtering, and category filtering
natively, reducing researcher cognitive load from 5 tools to 4.

## Observations

**What worked well:**

- First meta-research iteration run through the team skill -- worked as well as empirical
  runs. The compound Technical framing with 3 angles mapped naturally to the 4 sub-questions.
- Lead-dispatched follow-ups: zero nudging needed (4th consecutive success).
- Researchers independently discovered the same critical insight (Exa MCP limitations) from
  different angles -- genuine cross-validation.
- Researcher A did empirical testing in Round 2 (actually ran comparative Exa vs Firecrawl
  queries), adding real evidence beyond documentation review.
- Early shutdown of Researcher C was clean and efficient.
- Researcher B found significant blind spots in Round 2 (OpenAlex, Bluesky API, Firecrawl
  Cloudflare issues, underused Firecrawl formats) -- the follow-up was genuinely valuable.

**Process observations:**

- Meta-research through the team skill is viable and arguably better than manual research --
  the multi-angle approach and forced cross-referencing catch things manual research would miss.
- The decision to frame "search tool audit" as a Technical compound question worked well.
  Could serve as a template for future meta-research iterations.

**No problems found.** The skill ran smoothly across all phases.

## Changes Made

**5 changes to `researcher-prompt.md`:**

1. **Expanded "Available Tools" section** -- Added Firecrawl search to the tool list, added
   a "When to use which" decision guide distinguishing Exa (semantic discovery) from Firecrawl
   search (precision targeting with operators) from Firecrawl scrape (content extraction).

2. **Added "Firecrawl Search Operators" section** -- Table of Google-style operators (site:,
   exact match, exclusion, intitle:, inurl:, tbs date filtering with examples). Includes
   caveat about tbs filtering by index date vs publication date.

3. **Added "Academic Search" section** -- Three-step workflow (Exa for broad discovery ->
   Firecrawl search with site: for domain targeting -> Firecrawl scrape for full content).
   Lists useful academic domain targets. Lists free academic APIs (Semantic Scholar, OpenAlex,
   arXiv) with URL patterns.

4. **Expanded "Content Extraction" section** -- Added advanced scrape features: summary format
   for triage, links format for citation chains, JSON extraction with schema, includeTags/
   excludeTags, waitFor for SPAs, proxy options, PDF parsing, maxAge caching. Added "if scrape
   returns empty" workflow.

5. **Added "Accessing Restricted Content" section** -- Complete workaround guide: archive.ph
   for paywalls (with URL patterns), xcancel.com for Twitter/X, Reddit .json endpoint,
   Freedium for Medium, Bluesky public API, LinkedIn limitations. Each with exact URL patterns.

**1 additional change:** 6. **Updated Setup section** -- Added `ToolSearch: "+firecrawl search"`
to load Firecrawl search tool alongside scrape.

**Post-checkpoint fix:** Prettier mangled `>` characters in source evaluation priority chains
(e.g., `documentation > code examples > blog posts`) into markdown blockquotes. Replaced all
`>` separators with ", then" phrasing across all 6 type-specific entries for prettier safety.

**Post-iteration infrastructure change (backlog #18 resolved):**
Switched Exa MCP from npm package (`exa-mcp-server`) to hosted HTTP endpoint
(`mcp.exa.ai/mcp`) with `web_search_advanced_exa` and `people_search_exa` enabled. This
exposed all parameters identified as missing during Iteration 4 research. Firecrawl search
was then removed from the researcher prompt entirely -- Exa advanced covers domain targeting,
date filtering, and category filtering natively. Net effect: researchers now have 4 tools
instead of 5, simpler decision-making, and more powerful search capabilities.

**Post-iteration bug discovery (contextMaxCharacters suppresses summaries/highlights):**
Investigated why `enableHighlights` and `enableSummary` produced no visible output through
the Exa MCP endpoint. Read the npm package source and found two bugs in the response
formatter: (1) When `contextMaxCharacters` is set, the API returns a `context` field that
the formatter uses exclusively, dropping per-result data including summaries and highlights.
(2) The `ExaSearchResult` type doesn't include `highlights` at all -- they're requested from
the API but silently dropped. **Fix:** Use `textMaxCharacters: 1` + `enableSummary: true`.
Do NOT set `contextMaxCharacters`.

**Post-iteration tool simplification:**
Reduced Exa tools from 5 to 2: `web_search_advanced_exa` + `get_code_context_exa`.

**Post-iteration category guidance:**
Added category restriction table and usage hints from Exa skills repo.

## Notes for Next Iteration

- Researcher-prompt.md is ~420 lines. Monitor for issues.
- Default Exa search pattern (`enableSummary: true` + `textMaxCharacters: 1`) untested
  in empirical run. First test will reveal whether researchers follow the guidance.
- Category filter restrictions documented but untested.
- Academic search strategies documented but untested in Scientific/Health run.

# Iteration 5 -- Scientific/Health Empirical (IF + Cognitive Performance)

## Plan

**Type:** Empirical (Scientific/Health question)
**Backlog item:** #12
**Scope:** Broad (3 sonnet researchers, 2 rounds)

**Goal:** Test the evidence pyramid synthesis template, academic search guidance, and the
new Exa `enableSummary` + `textMaxCharacters: 1` default pattern. Also test whether the
category filter restrictions table prevents 400 errors. Secondary goal: validate that the
architecture continues to work smoothly (6th consecutive run).

**Question:** "Does intermittent fasting improve cognitive performance?"

**Angles:**
- (A) Systematic reviews and meta-analyses (anchor evidence)
- (B) Mechanistic pathways (BDNF, neuroinflammation, ketones, gut-brain axis)
- (C) Real-world protocols, practitioner evidence, and cognitive risks

## Research Summary

**Report:** `~/.claude/research/intermittent-fasting-cognitive-performance/report.md`

Round 1: All 3 researchers completed successfully. Researcher A found the Bamberg & Moreau
2025 meta-analysis (63 studies, 3,484 participants) as the anchor -- acute fasting is
cognitively neutral. Researcher B mapped 5 mechanistic pathways with 13 sources.
Researcher C covered protocols, null results, and disordered eating risks with 11 sources.

Triage identified: (1) CCR vs IF conflict (Researcher A's O'Leary review claimed CCR
superior, but other evidence was more nuanced), (2) population-specificity gap (which
populations benefit most?).

Round 2: Researcher B (fresh perspective) resolved the CCR vs IF conflict -- O'Leary
conflated fasting types; head-to-head trials show no difference when calorie-matched.
Researcher C (continuity) deep-dived population specificity -- metabolically impaired
populations benefit most, age is a moderator, ApoE4 data is missing.

Final report: ~300 lines, 26 footnotes, Scientific/Health evidence pyramid template.

**Key conclusions (calibrated confidence):**
- Acute fasting is cognitively neutral in healthy adults (highly likely, 80-92%)
- Chronic IF evidence is insufficient to claim cognitive benefit (highly likely)
- Metabolically impaired populations benefit most (likely, 63-79%)
- IF ≈ CCR when calorie-matched (likely)
- Mechanistic pathways are plausible but human translation is incomplete (likely)

## Observations

**What worked well:**

- Evidence pyramid template worked excellently.
- Lead-dispatched follow-ups: 6th consecutive success, zero nudging needed.
- Academic source quality was strong (APA, Cell Metabolism, Nature, BMJ Gut, Springer, MDPI).
- Triage cross-referencing caught genuine issues.
- Calibrated confidence language worked naturally.
- All researchers completed on first try.

**Exa parameter compliance:** 32/32 Exa calls fully compliant with summary pattern. Categories
used: `research paper` (frequent), `personal site` (once). No 400 errors. 28 Firecrawl scrape
calls, all with `onlyMainContent: true`.

## Changes Made

**3 changes (implementing backlog #23 -- researcher shutdown feedback reports):**

1. Updated Shutdown section in `references/researcher-prompt.md` with feedback file template.
2. Added Step 2 to Phase 10 (Cleanup) in `SKILL.md` for reading feedback files.
3. Added `scripts/analyze-transcripts.py` for post-hoc JSONL analysis.

**Backlog items resolved:** #12 (Scientific/Health empirical), #23 (shutdown feedback reports).

## Notes for Next Iteration

- Feedback reports implemented but untested. Next run will validate.
- Researcher-prompt.md is 438 lines. Monitor.
- Evidence pyramid validated. No changes needed.
- Exa guidance: 32/32 compliant. No further investment needed.
- Category restrictions untested with restricted categories (tweet, company, people).

# Iteration 6 -- Emerging/Frontier Empirical (LLM Reasoning)

## Plan

**Type:** Empirical (Emerging/Frontier question)
**Backlog item:** #9
**Scope:** Comprehensive (4 opus researchers, 3 rounds incl. verification)

**Goal:** Test the last major untested synthesis template (Emerging/Frontier with instability
warnings, date-sensitive source evaluation). First run with shutdown feedback reports
(implemented in Iteration 5 but untested). Exercise Exa categories beyond `research paper`
(`news`, `personal site`, `tweet`), testing the category restriction table.

**Question:** "What are the current approaches to LLM reasoning?"

**Angles:**

- (A) Foundational paradigms: CoT, ToT, GoT, and their successors
- (B) Training-time reasoning: PRM, RL, GRPO, RLVR, reasoning-focused training
- (C) Inference-time compute scaling: search, verification, test-time compute allocation
- (D) Emerging approaches: neurosymbolic, multi-agent, tool-augmented, latent reasoning

## Research Summary

**Report:** `~/.claude/research/llm-reasoning-approaches/report.md`

**Round 1:** All 4 opus researchers completed successfully. Strong convergence on RLVR as
dominant paradigm shift (A, B, C). Latent reasoning identified as frontier by A and D
independently. B provided deep GRPO/RLVR coverage. C mapped inference-time compute scaling
with strong evidence. D covered neurosymbolic, multi-agent, tool-augmented, latent, and
world model approaches. Total: ~60 sources across researchers.

**Triage identified:** (1) PRM vs ORM conflict (B says PRMs disappoint at scale; C says PRMs
outperform), (2) all 4 researchers noted heavy academic bias -- practitioner/deployment
perspectives missing, (3) competitive landscape gap.

**Round 2:** Researcher A (fresh perspective) resolved PRM conflict -- both correct in different
regimes (inference-time verification: PRM wins; large-scale RL training: ORM preferred).
Researcher D (gap-fill) surveyed practitioner perspectives and competitive landscape using
diverse Exa categories (news, personal site, tweet).

**Round 3 (Verification):** Researcher B verified CoT faithfulness claims -- 2.3% True Thinking
Score CONTESTED (single model/dataset, not replicated; direction supported but specific number
misleading), <20% verbalization SUPPORTED WITH NUANCE (multiple independent confirmations;
METR argues manageable for safety monitoring).

**Final report:** ~400 lines, 55 footnotes. Emerging/Frontier template with instability warnings.

## Observations

**What worked well:**

- **Emerging/Frontier template worked excellently.** The "Current State (as of date)" +
  "Trajectory" + "Instability Warning" structure produced a report that acknowledges the
  rapidly evolving nature of the field while still providing definitive findings.
- **Lead-dispatched follow-ups:** 7th consecutive success, zero nudging needed.
- **Cross-agent verification added genuine value.** The CoT faithfulness verification produced
  a nuanced CONTESTED verdict that materially improved the report -- the 2.3% number was
  tempered to a directional finding.
- **PRM conflict resolution was clean.** Researcher A (fresh perspective) found the resolution
  along the inference vs training axis with 13 independent sources.
- **All researchers completed on first try.** No stalls, no nudging, no retries.
- **Category restriction table worked.** `tweet` category used successfully (no 400 errors).
  Researcher D correctly omitted restricted params for tweet queries.

**Exa parameter compliance (verified via `scripts/analyze-transcripts.py`):**

- **49/49 Exa calls (100%) fully compliant.** `enableSummary: true` + `textMaxCharacters: 1`,
  zero `contextMaxCharacters` violations. 7th session with perfect compliance (cumulative
  81/81 across Iterations 5-6).
- **Categories exercised:** `research paper` (8), `news` (2), `personal site` (2), `tweet` (1).
  First run using `news`, `personal site`, and `tweet` categories. No 400 errors.
- **Firecrawl:** 22 scrape calls, all `firecrawl_scrape`. No map or other tools used.
- **Rich parameter usage:** `startPublishedDate` widely used, `numResults` varied (5-10),
  `includeDomains` used once (metr.org).

**Shutdown feedback reports (first test -- backlog #23 validated):**

All 4 researchers wrote feedback files. Key findings from feedback:

- **arxiv scraping noise:** 3 of 4 researchers independently flagged that Firecrawl scraping
  of arxiv abstract pages returns mostly navigation boilerplate. All recommended using
  `arxiv.org/html/` or relying on Exa summaries for triage.
- **Tweet category thinness:** Researcher D noted tweet summaries were 1-2 sentences with
  limited context. The category works but provides lower signal density.
- **Duplicate self-referential messages:** 3 of 4 researchers reported receiving their own
  task assignment messages echoed back. Cosmetic but confusing. Likely a platform bug.
- **Verification verdict granularity:** Researcher B noted the SUPPORTED/CONTESTED/UNCHANGED
  trichotomy doesn't cover partial support. Iteration 2 verifiers also used "SUPPORTED WITH
  NUANCE."
- **Conflict resolution length:** Researcher A noted ~120-line target is too short for
  conflict resolution tasks that need comparison tables.
- **Positive:** All researchers praised the `enableSummary` + `textMaxCharacters: 1` pattern
  for efficient triage.

**Feedback reports vs transcript analysis comparison:**

Both mechanisms add unique value. Transcripts provide exact parameter compliance (definitive).
Feedback reports provide qualitative insights not visible in transcripts: arxiv scraping issue,
duplicate message bug, tweet category thinness, verification verdict suggestions. **Keep both.**

## Changes Made

**No skill file changes this iteration.** The Emerging/Frontier template, feedback reports,
and category restriction table all validated successfully. Two quick changes identified for
next iteration (backlog #26 arxiv guidance, #27 verification verdict expansion) but not
blocking.

**Backlog items resolved:** #9 (Emerging/Frontier empirical), #23 validated (feedback reports
work as designed).

## Notes for Next Iteration

- **All 7 synthesis templates now tested.** Factual, Scientific/Health, Consumer, Technical,
  Opinion/Sentiment (untested), Contested, and Emerging/Frontier. Only Opinion/Sentiment
  remains untested, but it's the least complex template. The skill is mature for synthesis.
- **Exa compliance is thorough.** 81/81 across 2 measured iterations. No further monitoring
  investment needed -- the guidance works reliably.
- **Feedback reports validated and useful.** They surfaced 4 actionable findings this iteration.
  Worth keeping as standard practice.
- **Quick wins available:** Backlog #26 (arxiv guidance) and #27 (verification verdict
  expansion) are small, targeted changes that can be done without a full research run.
- **Recommended next:** Backlog #25 (audit against Anthropic agent teams docs) is a quick
  meta task that doesn't require running the skill. Good for a shorter iteration.

# Iteration 7 -- Agent Teams Docs Audit (Meta)

## Plan

**Type:** Meta-research (docs audit)
**Backlog item:** #25
**Scope:** No research run -- audit existing skill against official Anthropic documentation.

**Goal:** Read official Claude Code agent teams docs and compare our implementation against
recommended patterns, known limitations, and best practices. Use Exa for discovery and
Firecrawl for scraping (not the claude-code-guide agent). Also implement quick wins #26
(arxiv guidance) and #27 (verification verdict expansion).

## Sources Scraped

6 official pages scraped with Firecrawl (`onlyMainContent: true`, `maxAge: 86400000`):

1. `code.claude.com/docs/en/agent-teams` -- Primary agent teams documentation
2. `code.claude.com/docs/en/sub-agents` -- Subagents documentation (for comparison)
3. `platform.claude.com/docs/en/agent-sdk/subagents` -- SDK subagents (programmatic API)
4. `code.claude.com/docs/en/best-practices` -- Best practices guide
5. `anthropic.com/engineering/multi-agent-research-system` -- Anthropic's own research system
6. `claude.com/blog/building-multi-agent-systems-when-and-how-to-use-them` -- Multi-agent guide

Plus 3 Exa searches for discovery (30 results total across `docs.anthropic.com`,
`github.com/anthropics`, and general web).

## Audit Findings

**Green Flags (architecture confirmed correct):**

- **Hub-and-spoke / orchestrator-worker pattern** matches both the agent teams docs and
  Anthropic's own research system architecture. The docs explicitly describe "one session acts
  as the team lead, coordinating work, assigning tasks, and synthesizing results."
- **No peer-to-peer researcher communication** aligns with the agent teams docs' model where
  the lead coordinates all work. The multi-agent blog emphasizes "context-centric decomposition"
  and warns against the "telephone game" of inter-agent information loss.
- **Lead-dispatched assignments** validated. Docs support both explicit assignment and self-claim
  with file locking, but our empirical finding (Iterations 1-2) that lead-dispatched works
  better is consistent with Anthropic's advice to "teach the orchestrator how to delegate."
- **Filesystem output** explicitly recommended. Anthropic's research blog says: "Subagent output
  to a filesystem to minimize the 'game of telephone'... subagents call tools to store their
  work in external systems, then pass lightweight references back to the coordinator."
- **Verification subagent pattern** matches the blog's "verification subagent pattern" exactly,
  including our "assign to fresh perspective" approach.
- **Context-centric decomposition by research angle** (not by role) matches the blog's strong
  recommendation against problem-centric splitting.
- **Shutdown then TeamDelete sequence** matches docs' requirement that cleanup fails if active
  teammates remain.
- **No nested teams** -- our researchers don't spawn sub-teams, which is correct since the docs
  confirm "teammates cannot spawn their own teams."
- **Automatic message delivery, no polling** -- confirmed by docs.

**New Information (resolved questions):**

- **Permission inheritance confirmed:** "Teammates start with the lead's permission settings.
  If the lead runs with `--dangerously-skip-permissions`, all teammates do too." Can change
  individual modes after spawning, but not at spawn time. This partially resolves backlog #5.
- **CLAUDE.md + MCP servers inherited automatically:** "When spawned, a teammate loads the same
  project context as a regular session: CLAUDE.md, MCP servers, and skills." This explains why
  researchers can use ToolSearch to load Exa/Firecrawl -- they inherit MCP server access.
- **Task dependencies available:** Framework supports `blockedBy` for task ordering. We don't
  use this (our lead-dispatched approach handles sequencing manually), but it's available if
  we ever need it.
- **Session resumption doesn't restore teammates:** Our `state.md` resume check in Phase 0 is
  important precisely because of this limitation. Good that we built this.
- **Delegate mode exists:** Pressing Shift+Tab restricts the lead to coordination-only tools.
  Not needed for our skill (which operates programmatically) but useful to know.

**Anthropic's research system comparison:**

Their architecture is strikingly similar to ours. Key parallels: orchestrator-worker pattern,
lead decomposes query and spawns subagents for different facets in parallel, subagents search
independently then return distilled findings, lead synthesizes. Their system uses Opus 4 lead +
Sonnet 4 subagents and outperformed single-agent by 90.2% on their internal eval.

Key difference: they found "token usage by itself explains 80% of the variance." Their effort
scaling rules: simple queries get 1 agent with 3-10 tool calls, direct comparisons 2-4 agents
with 10-15 calls each, complex research 10+ agents. Our scale is more conservative (max 4
researchers for Comprehensive) but appropriate for our token budget.

They also use a separate CitationAgent for adding citations to the final output. We don't have
this -- our lead handles synthesis and citations together. Worth monitoring but not needed now.

**No anti-patterns found.** Our architecture doesn't fight the framework, uses no deprecated
patterns, and follows the recommended cleanup sequence.

**Yellow flags (minor, non-blocking):**

- Our Comprehensive scope uses Opus for all researchers. Anthropic's own system uses Sonnet
  workers successfully. Consider testing Sonnet researchers in Comprehensive to reduce cost
  while maintaining quality. Added as backlog #28.
- No explicit TaskList polling during long researcher waits. If a researcher silently crashes,
  the lead would wait indefinitely. Low risk (hasn't happened in 7 iterations) but worth
  awareness.

## Changes Made

1. **[#26] arxiv scraping guidance** -- Added to `references/researcher-prompt.md` under
   Content Extraction: guidance to use `arxiv.org/html/{id}` instead of `/abs/`, prefer Exa
   summaries for triage, use arxiv API for structured extraction.

2. **[#27] Verification verdict expansion** -- Added SUPPORTED WITH NUANCE as fourth verdict
   in SKILL.md Phase 7, with description: "Directionally correct but specific details differ
   or require qualification."

**Backlog items resolved:** #25 (docs audit), #26 (arxiv guidance), #27 (verification verdict).

## Notes for Next Iteration

- Architecture is validated and mature. 7 iterations without a major structural change needed.
- Permission setup (#5) is now better informed: inheritance confirmed, specific permissions
  needed are mkdir + Write to research dir + Bash.
- Scope calibration (#6) has 7 data points and strong evidence from Anthropic's own scaling
  rules to compare against.
- Consider testing mixed model config: Opus lead + Sonnet researchers for Comprehensive (#28).

# Iteration 8 -- Scope Calibration Refinement (Meta)

## Plan

**Type:** Meta-analysis (internal data synthesis)
**Backlog item:** #6
**Scope:** No research run -- analyze 7 empirical data points + Anthropic's scaling rules to
refine scope calibration guidance.

**Goal:** Determine whether the type-to-scope defaults need adjustment, add escalation and
de-escalation signals, and add empirical output expectations. Also subsumes backlog #8 (report
length targets).

## Analysis

**Data compiled from 7 runs:**

| Iter | Type               | Scope         | Rounds Used | Sources | Lines | Round 2 Value         | Assessment     |
| ---- | ------------------ | ------------- | ----------- | ------- | ----- | --------------------- | -------------- |
| 1    | Technical+Consumer | Broad         | 2           | 25      | ~250  | Useful                | Right-sized    |
| 2    | Contested          | Comprehensive | 3           | 60+     | ~350  | Useful                | Right-sized    |
| 3    | Consumer           | Broad         | 2           | 25      | ~200  | Useful-not-essential  | Slight overfit |
| 4    | Technical (meta)   | Broad         | 2           | 24      | ~280  | Useful                | Right-sized    |
| 5    | Scientific/Health  | Broad         | 2           | 26      | ~300  | Useful                | Right-sized    |
| 6    | Emerging/Frontier  | Comprehensive | 3           | 55      | ~400  | Useful (conflict+gap) | Right-sized    |
| 9    | Opinion/Sentiment  | Comprehensive | 2           | 60+     | ~370  | Useful (all 4 gaps)   | Right-sized    |

**Key findings:**

1. **Current defaults are correct 6/6 times.** Consumer was the only case with slack (Round 2
   useful-not-essential). All other scopes were right-sized.

2. **Consumer is the only type with de-escalation evidence.** When the product category is
   narrow, well-reviewed, and expert consensus is clear, Focused could suffice. Iteration 3
   (keyboards) is the data point.

3. **Comprehensive verification (Round 3) added value both times tested.** Iteration 2:
   SUPPORTED WITH NUANCE + CONTESTED verdicts. Iteration 6: CONTESTED + SUPPORTED WITH NUANCE.
   Both materially improved the final report.

4. **Focused scope has never been tested.** Only Factual defaults to it, and no Factual question
   has been run. Output expectations for Focused are projected from Round 1 subsets of Broad
   runs.

5. **Anthropic comparison:** Their simple (1 agent, 3-10 calls) maps to "don't use this skill."
   Their comparisons (2-4 agents) map to our Focused/Broad. Their complex (10+ agents) exceeds
   our maximum. Our system is well-sized for the middle range.

6. **Report length is scope-driven, not type-driven.** The variation across types within Broad
   (200-300 lines) is smaller than the variation across scopes. So output expectations belong on
   the scope tier, not the question type.

## Changes Made

1. **Scope modifiers added to `references/question-types.md`** -- Per-type de-escalation and
   escalation signals, with the Consumer de-escalation backed by Iteration 3 evidence. Also
   added "do not use this skill" criteria for simple queries.

2. **Empirical output expectations added to `references/question-types.md`** -- Table of
   report length and source count ranges per scope tier, based on 7 runs. Focused estimates
   projected. Explicitly noted these are descriptive ranges, not targets.

3. **SKILL.md Phase 2 updated** -- Now references scope modifiers from question-types.md
   instead of inlining ad-hoc adjustment criteria.

**Backlog items resolved:** #6 (scope calibration), #8 (report length targets).

## Notes for Next Iteration

- **Focused scope remains untested.** A Factual question run would validate the Focused tier
  and the projected output expectations, but it's low priority since the skill explicitly says
  "don't use for simple factual lookups."
- **The main remaining empirical gap is Opinion/Sentiment** -- the last untested synthesis
  template. A good candidate for the next empirical run.
- **Mixed model config (#7) is the highest-value empirical test** -- combining it with an
  Opinion/Sentiment question would test two things at once.

# Iteration 9 -- Opinion/Sentiment + Mixed Model Config (Empirical)

## Plan

**Type:** Empirical (Opinion/Sentiment question)
**Backlog items:** #7 (mixed model config), last untested synthesis template
**Scope:** Comprehensive (4 sonnet researchers, 2 rounds -- model override from opus to sonnet)

**Goal:** Combine two tests in a single empirical run: (1) Validate Sonnet researchers for
Comprehensive scope (backlog #7), replacing the Opus default. Anthropic's own research system
uses Opus lead + Sonnet subagents. (2) Test the Opinion/Sentiment synthesis template, the last
of 7 templates never used in a real run.

**Question:** "What do developers actually think about Nix and NixOS in 2026?"

**Angles:**

- (A) Community sentiment: overall vibe, praise vs frustration, how feelings have shifted
- (B) Pain points and barriers: learning curve, documentation, Nix language, flakes controversy
- (C) Enterprise/professional adoption: companies using Nix, CI/CD patterns, onboarding stories
- (D) Governance and community health: foundation drama, forks (Lix, Aux, Tvix), SC elections

## Research Summary

**Report:** `~/.claude/research/developer-opinions-nix-nixos-2026/report.md`

**Round 1:** All 4 Sonnet researchers completed successfully. Strong convergence on the
paradox theme: fastest community growth ever (30% YoY) alongside deepest fragmentation ever
(governance crisis, three forks). Researcher A surfaced the sentiment spectrum from "Nix
changed my life" to "I quit after 6 months." Researcher B documented the learning curve and
documentation as top pain points across all sources. Researcher C found enterprise adoption
concentrated in devshells-only patterns with wrapper tools. Researcher D provided deep coverage
of the 2024-2025 governance crisis and its aftermath. Total: ~50+ sources across researchers.

**Triage identified:** (1) Gap: moderate/ambivalent voices underrepresented (survivorship bias
in online discussions), (2) DetSys conflict of interest not fully explored (company behind
flakes also behind commercial products), (3) wrapper tools ecosystem deserved deeper
comparison, (4) Steering Committee election results and their impact not covered.

**Round 2:** Researcher A (gap-fill) found moderate voices -- "Nix purgatory" users stuck
between love and frustration, plus successful teams who don't post online. Researcher B
(deep-dive) investigated DetSys conflict with nuance -- found community resentment but also
genuine engineering contributions. Researcher C (deep-dive) did comprehensive wrapper tools
comparison with GitHub stars, feature matrices, and community adoption data. Researcher D
(gap-fill) covered SC election results and post-crisis governance trajectory.

**Round 3 skipped:** No single-source high-impact claims remaining. All major conflicts
resolved by Round 2. Remaining uncertainties about magnitude, not direction.

**Final report:** ~370 lines, 50 footnotes, 60+ sources. Opinion/Sentiment template with
5-segment distribution of views and representativeness assessment.

## Observations

**What worked well:**

- **Opinion/Sentiment template validated with adaptation.** The 3-segment default structure
  (Majority/Minority/Outlier) was adapted to 5 segments to capture the nuanced reality:
  Passionate Core (~15-20%), Pragmatic Middle (~30-35%), Frustrated Dropouts (~20-25%),
  Nix Purgatory (~15-20%), and Broader Dev Population (~10-15%). The template's
  Representativeness Assessment section added genuine value.
- **Sonnet researchers produced Comprehensive-quality output.** All 4 researchers completed
  both rounds, followed all methodology guidance, wrote structured findings and feedback
  reports. No evidence of quality degradation vs Opus researchers in Iterations 2 and 6.
- **Lead-dispatched follow-ups:** 8th consecutive success, zero nudging needed.
- **All researchers completed on first try.** No stalls, no nudging, no retries.
- **Feedback reports written by all 4 researchers.** Continued validation of this mechanism.
- **Verification skip was correct.** Round 2 resolved all major gaps and conflicts. No
  single-source claims would have changed the report's conclusions.

**Permission friction observed:**

- **curl commands required user approval.** Researchers used `curl` for Reddit JSON API and
  GitHub API endpoints, triggering ~8-10 permission prompts during the run. This is the first
  time permission friction was significant enough for the user to raise it post-run.
- **Root cause:** The researcher-prompt.md Reddit section recommends curl for Reddit JSON API
  (since Exa doesn't index Reddit and Firecrawl blocks it). GitHub API was also accessed via
  curl for stars/forks data.

**Researcher feedback highlights:**

- Researcher A: Exa with enableSummary + textMaxCharacters:1 excellent for rapid triage.
  Category gotchas (tweet, company) useful to know upfront.
- Researcher B: Firecrawl scraping of HN threads returned full comment trees in markdown --
  very useful for community sentiment capture.
- Researcher C: Reddit JSON API `restrict_sr=on` was essential -- without it, global results
  instead of subreddit-specific. Exa didn't index GitHub issue bodies well. Recommended
  GitHub search API patterns in methodology.
- Researcher D: Strong coverage of Discourse-based forums (NixOS Discourse) via direct
  Firecrawl scraping. Exa worked well for blog posts and personal sites.

## Changes Made

3 changes to `references/researcher-prompt.md`:

1. **Reddit section updated** -- Added `restrict_sr=on` requirement with example URL pattern,
   guidance to prefer `/top/` and `/hot/` over `/search/` for sentiment research (browsing
   reveals organic discussions), note that curl requires user permission approval each time.

2. **GitHub data guidance added** -- New section before "Medium paywalled articles": Exa
   `get_code_context_exa` as primary tool for GitHub content discovery, Firecrawl for deep
   extraction of specific pages, `gh api` for quantitative data (stars, forks, issues) since
   it's already permitted. Explicit "do not use curl for GitHub API endpoints."

3. **Discourse JSON search API added** -- New section after GitHub: Firecrawl scrape of
   `/search.json?q=...` as fallback when Exa can't find specific forum topics. Returns
   structured results with topic IDs for follow-up scraping.

**Backlog items resolved:** #7 (mixed model config -- Sonnet validated for Comprehensive).

## Notes for Next Iteration

- **All 7 synthesis templates now tested.** The skill is mature for research operations.
  Opinion/Sentiment required the most template adaptation (5 segments vs 3) but the framework
  accommodated this well.
- **Sonnet validated as viable Comprehensive override, Opus kept as default.** One successful
  run (Opinion/Sentiment) doesn't prove Sonnet handles the hardest Comprehensive questions.
  SKILL.md notes Sonnet as an available override when cost matters.
- **Reddit MCP is the top engineering priority.** The curl permission friction this iteration
  is the strongest signal yet. Backlog #8 has full context on options.
- **Permission setup (#5) remains important** for shareability. Inheritance is confirmed but
  specific Bash permissions (curl, sleep) still cause friction.

# Iteration 10 -- Reddit MCP Server Integration (Engineering)

## Plan

**Type:** Engineering (no research run)
**Backlog item:** #8 (Reddit MCP server integration)

**Goal:** Eliminate the last curl dependency in researcher workflows by integrating a Reddit
MCP server via 1MCP, and updating the researcher methodology to use native MCP tools instead
of curl for Reddit content access.

**Background:** Iteration 9 confirmed significant permission friction (~8-10 curl prompts per
run) from Reddit JSON API access. Prior research (`~/.claude/research/reddit-access-for-claude-code/report.md`)
identified `jordanburke/reddit-mcp-server` as the top option. Fresh evaluation confirmed:
- Still actively maintained (v1.2.1 published ~10 days ago, 108 commits, 4 contributors)
- 9 read tools + 6 write tools, including `search_reddit` (critical for researchers)
- Anonymous mode: zero credentials, ~10 rpm, zero setup
- `npx reddit-mcp-server` -- no install needed, 1MCP compatible via stdio
- Runner-up (Hawstein/mcp-server-reddit) is stale -- no commits in 10 months, no search tool
- New entrants (liuyang1520, karanb192) too immature (0 stars, requires auth/local install)

Legal context unchanged: Reddit v. Anthropic lawsuit makes this fraught, but low-volume
personal use risk is low. Anonymous mode doesn't even use API credentials.

## Changes Made

### 1. Added Reddit MCP server to 1MCP config

**File:** `configs/claude/1mcp.json`

Added `reddit` server entry with `npx reddit-mcp-server` and explicit `REDDIT_AUTH_MODE: "anonymous"`.
Placed alphabetically before the `gws` entry. Server will be available via 1MCP as
`mcp__1mcp__reddit_1mcp_{tool_name}`.

### 2. Updated researcher methodology -- Setup section

**File:** `~/.claude/skills/deep-research-team/references/researcher-prompt.md`

Added `ToolSearch: "+reddit"` to the tool loading instructions in the Setup section.

### 3. Updated researcher methodology -- Available Tools section

**File:** same

Added **Reddit** tool block documenting 4 content extraction tools (one-line descriptions,
params left to MCP schemas): `get_top_posts`, `get_post_comments`, `get_reddit_post`,
`get_subreddit_info`. Deliberately excluded `search_reddit` -- Google's Reddit index (via
Firecrawl search) is far superior to Reddit's native search.

Added to "When to use which": Firecrawl search for Reddit discovery (`site:reddit.com`),
Reddit MCP tools for content extraction.

### 4. Replaced curl-based Reddit access with MCP tools

**File:** same, "Accessing Restricted Content > Reddit content" section

Completely rewrote the Reddit content section. Removed all curl-based guidance. Replaced with
two-layer pattern: Firecrawl search (`site:reddit.com {query}`) for discovery, Reddit MCP
tools for content extraction. Preserved the key insight that `get_top_posts` is preferred over
search for sentiment research.

## Impact

- **Permission friction:** Eliminates ~8-10 curl permission prompts per research run that
  accesses Reddit. Reddit access now uses native MCP tools that require no special permissions.
- **Researcher experience:** Researchers use the same tool pattern (ToolSearch -> MCP call)
  for Reddit as they do for Exa and Firecrawl. No more shelling out to curl.
- **Coverage:** All previously curl-accessible Reddit features are covered by MCP tools, plus
  additional features (engagement analysis, subreddit stats, trending subreddits).
- **curl dependency:** With GitHub curl eliminated in Iteration 9 and Reddit curl eliminated
  here, researchers should have zero curl dependencies in normal workflows. The only remaining
  curl use cases are the Bluesky public API and rate-limit retry sleeps (which use Bash `sleep`,
  not curl).

## Backlog Resolution

**#8 resolved.** Reddit MCP server integrated via 1MCP with anonymous mode. Researcher
methodology updated to use native MCP tools. curl-based Reddit access guidance removed.

# Iteration 11 -- Reddit MCP Empirical Validation

## Type

Empirical (Focused scope run to validate Iteration 10 engineering changes)

## Question

"What is the best community-favorite content about the TV show Dark -- videos, write-ups,
essays, analyses, explainers, fan theories?"

## Config

- **Scope:** Focused (2 researchers, 1 round)
- **Question type:** Consumer/Recommendation
- **Model:** Sonnet researchers
- **Report:** `~/.claude/research/dark-tv-show-community-content/report.md`

## Validation Goals & Results

| Goal                                          | Result | Notes                                                                                       |
| --------------------------------------------- | ------ | ------------------------------------------------------------------------------------------- |
| Researchers load Reddit tools via ToolSearch   | Pass   | Both loaded successfully                                                                    |
| Firecrawl search with site:reddit.com works   | Pass   | Both used it. B called it "the single most valuable technique"                              |
| get_top_posts extracts Reddit content          | Pass   | Both used. Returns memes/appreciation at top; insufficient alone for substantive threads     |
| get_post_comments extracts thread discussions  | Pass   | A used on 4 threads, B on 2. Specific upvote counts, ratios, user quotes extracted          |
| get_reddit_post works for specific posts       | Pass   | B used on 7 posts with engagement metrics                                                   |
| get_subreddit_info works                       | Pass   | B used to get 213K member count                                                             |
| search_reddit works                            | Pass   | A used 2 queries                                                                            |
| Zero curl usage for Reddit                     | Pass   | No permission prompts observed                                                              |
| Useful findings produced                       | Pass   | 24-source report with specific engagement data, 5 categories of content                     |

**Overall: 9/9 validation goals passed. Reddit MCP integration works end-to-end.**

## Tool Usage Summary

### Researcher A

- **Exa:** enableSummary, textMaxCharacters, numResults (standard pattern)
- **Firecrawl:** firecrawl_search for Reddit discovery
- **Reddit MCP:** search_reddit (2), get_top_posts (1), get_post_comments (4)
- **Issues:** `site:reddit.com/r/DarK` in Firecrawl search was too specific, returned generic
  results. Broader `site:reddit.com` worked better.

### Researcher B

- **Exa:** enableSummary, textMaxCharacters, numResults, excludeDomains, category (personal site)
- **Firecrawl:** firecrawl_scrape (4 URLs), firecrawl_search for Reddit discovery
- **Reddit MCP:** get_top_posts (1), get_post_comments (2), get_reddit_post (7), get_subreddit_info (1)
- **Issues:** Medium paywall blocked full markdown extraction (summary format worked). Top posts
  from r/DarK were overwhelmingly memes -- needed Firecrawl search for substantive threads.

## Key Observations

### Two-layer Reddit pattern validated

The Firecrawl search (discovery) + Reddit MCP (extraction) pattern works well. Both researchers
used it independently and both endorsed it in feedback.

### get_top_posts is necessary but insufficient

For topic-specific threads, `get_top_posts` returns the most popular posts overall (which for
entertainment subreddits are memes and appreciation posts, not analytical content). The Firecrawl
search layer is essential for finding substantive threads on specific topics.

### Firecrawl search scoping

Researcher A found that `site:reddit.com/r/DarK` (subreddit-scoped) returned generic video
essay threads instead of r/DarK-specific content. The broader `site:reddit.com` with topic
keywords in the query worked better. This is worth noting in the methodology.

### Researcher complementarity

Good angle separation produced minimal overlap. Both independently found the mmmmmmmmichaelscott
FAQ and The_Wattsatron easter eggs post (confirming canonical status via independent discovery).
A focused on video/audio content; B focused on written/academic content.

### Exa category filter usage

Researcher B used `category: "personal site"` to filter to independent blog essays, which
effectively surfaced philosophical analyses while filtering out corporate listicle content.
Good technique for this question type.

## Changes Made

### 1. Note Firecrawl search scoping in researcher-prompt.md

Add note that `site:reddit.com` (broad) works better than `site:reddit.com/r/subreddit`
(subreddit-scoped) in Firecrawl search queries.

### 2. Note get_top_posts limitation in researcher-prompt.md

Clarify that `get_top_posts` returns the most popular posts overall (often memes/meta for
entertainment subreddits), not necessarily the most substantive analytical content. Recommend
using Firecrawl search for finding topic-specific substantive threads.

## Transcript Analysis

Script: `analyze-transcripts.py --session 5213f595-f4b6-4140-b7e0-5d2908d822d7`

**Exa compliance: 10/10 (100%)**
- All calls used `enableSummary: true` + `textMaxCharacters: 1`
- Zero `contextMaxCharacters` violations

**All MCP calls (42 total across both researchers):**
- Reddit: 22 (get_reddit_post: 9, get_post_comments: 7, search_reddit: 3, get_top_posts: 2, get_subreddit_info: 1)
- Exa: 10 (web_search_advanced_exa: 10)
- Firecrawl: 10 (firecrawl_search: 6, firecrawl_scrape: 4)

**Notable patterns:**
- Researcher B used `excludeDomains: ["reddit.com"]` on Exa to avoid duplicate Reddit results
- Researcher B used `category: "personal site"` for blog essay discovery (2 calls)
- Reddit was the most-used MCP server — appropriate for a community-content question

## Post-Iteration Housekeeping

Changes made in the same session after the main iteration writeup:

### 3. Generalize analyze-transcripts.py to track all MCP tools

Replaced hardcoded Exa/Firecrawl parsing with generic MCP tool detection via
`re.compile(r"(?:mcp__1mcp__)?(\w+?)_1mcp_(\w+)")`. All 1MCP tools now captured and grouped
by server. Custom formatters for Exa (with compliance checking), Firecrawl, and Reddit.
Aggregate report shows per-server totals and per-endpoint breakdown.

### 4. Fix --project-dir to accept filesystem paths

`--project-dir` now accepts regular filesystem paths (e.g., `/Users/malo/.config/nix-config`)
in addition to `~/.claude/projects/{slug}` paths. Slug algorithm: `re.sub(r"[^a-zA-Z0-9]", "-", path)`.

### 5. Simplify researcher feedback template

Removed the "Search Tools" section (redundant now that the transcript script extracts all MCP
tool usage automatically). Reframed template as qualitative reflection: What Worked, Issues,
Suggestions. Added framing note so researchers understand the purpose.

## Backlog Changes

- No items resolved (validation run, not feature addition)

**Cumulative changes including housekeeping: 30 + 3 = 33.**

# Iteration 12 -- Firecrawl Scrapability Testing

**Type:** Engineering (no research team)
**Goal:** Empirically test all documented workaround URLs in researcher-prompt.md with actual
Firecrawl scrape calls. Remove or caveat workarounds that don't work.

## Test Results

### 1. archive.ph

**URL tested:** `https://archive.ph/newest/https://www.nytimes.com/2025/03/10/technology/ai-agents-future.html`
**Params tried:** `proxy: "stealth"`, `proxy: "enhanced"`, `waitFor: 5000`
**Result:** BROKEN. reCAPTCHA challenge page (status 429) on all attempts. Cloudflare protection
blocks Firecrawl completely. Costs 5 credits per attempt (stealth/enhanced proxy).
**Conclusion:** Remove as recommended workaround for Firecrawl-based research.

### 2. Wayback Machine

**URLs tested:**
- `web.archive.org/web/https://www.wsj.com/...` (shortcut format) -- 404, "not archived"
- `web.archive.org/web/2024/https://www.nytimes.com/...` (year only) -- 404, "not archived"
- `web.archive.org/web/2024*/https://arstechnica.com/...` (wildcard) -- returns calendar page
- `web.archive.org/web/20240515120000/https://en.wikipedia.org/wiki/GPT-4` (exact timestamp) -- **SUCCESS**, full content (149K chars, had to be saved to file)
- `web.archive.org/web/https://en.wikipedia.org/wiki/GPT-4` (shortcut, known-archived page) -- **SUCCESS**, 173K chars, status 200. Firecrawl followed redirect to `/web/20260203171033/...` (most recent snapshot). Wayback toolbar at top but full article content follows.
- `web.archive.org/web/2/https://en.wikipedia.org/wiki/GPT-4` ("/2/" trick) -- **SUCCESS**, same result as shortcut format.

**Result:** WORKS with simple shortcut format `/web/{URL}`. Initial tests with WSJ/NYTimes
shortcut URLs returned 404 because those pages weren't archived, not because the format was
wrong. Retesting with a known-archived page (Wikipedia GPT-4) confirmed the shortcut redirects
to the most recent snapshot. Wayback Machine toolbar is prepended to extracted content but full
page follows. 1 credit.
**Conclusion:** Simplify guidance -- just use `/web/{URL}`. No need for exact timestamps.
**Lesson:** Test URL format correctness against pages known to be archived.

### 3. xcancel.com (Twitter/X)

**URLs tested:**
- `xcancel.com/sama/status/1895533845455577446` (without proxy) -- anti-bot challenge (503)
- `xcancel.com/sama/status/1895533845455577446` (stealth + waitFor) -- "Tweet not found" (404)
- `xcancel.com/elikiowa/status/1880305905189106091` (stealth + waitFor) -- "Tweet not found" (404)
- `xcancel.com/sama/status/1890816782836904000` (stealth + waitFor, verified valid ID) -- **SUCCESS**
- `xcancel.com/OpenAI/status/1790070592011288831` (stealth + waitFor) -- "Tweet not found" (404)

**Result:** WORKS but requires `proxy: "stealth"` + `waitFor: 5000` (5 credits). Initial tests
with invalid tweet IDs gave misleading "Tweet not found" (404) results. Retests with verified
valid tweet IDs confirmed: without proxy, anti-bot challenge blocks (503); with stealth +
waitFor, full content returned (tweet text, replies, engagement metrics). Confirmed on both
cached and fresh (`maxAge: 0`) URLs to rule out caching artifacts.
**Conclusion:** Keep with updated guidance: requires stealth proxy + waitFor, costs 5 credits.
**Lesson:** Always verify test inputs before concluding a service is broken.

### 4. freedium.cfd (Medium)

**URL tested:** `https://freedium.cfd/https://towardsdatascience.com/rag-vs-fine-tuning-...`
**Result:** DEAD. DNS resolution failure -- domain no longer resolves.
**Conclusion:** Replace with freedium-mirror.cfd.

### 5. freedium-mirror.cfd (Medium -- replacement)

**URL tested:** `https://freedium-mirror.cfd/https://towardsdatascience.com/rag-vs-fine-tuning-which-is-the-best-tool-to-boost-your-llm-application-94654b1eaba7`
**Result:** SUCCESS. Full article content extracted (status 200, 1 credit, basic proxy).
Complete Medium article with formatting, images, author info, tags. No paywall.
**Conclusion:** Document as the working Medium paywall bypass.

### 6. Bluesky

**URLs tested:**
- `public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=nix+nixos&limit=5` (raw API) -- 403 Forbidden
- `bsky.app/profile/jay.bsky.team/post/3lihkfohpkk2t` (individual post, no waitFor) -- "Post not found" (SPA not rendered)
- `bsky.app/profile/bsky.app` (profile page, waitFor: 5000) -- **SUCCESS**, massive content (full profile + dozens of recent posts)
- `bsky.app/profile/atprotocol.dev` (profile page, waitFor: 5000) -- **SUCCESS**, 94K chars

**Result:** WORKS but `bsky.app` is a JavaScript SPA -- `waitFor: 5000` is required for content
to render. Without it, individual post pages return "Post not found". Profile pages with waitFor
return extensive content including all recent posts. No proxy needed (1 credit). The raw API
endpoint was unnecessary in the first place.
**Conclusion:** Remove API-specific guidance. Document that bsky.app needs `waitFor: 5000` (SPA).

### 7. firecrawl_search site:reddit.com

**Query tested:** `site:reddit.com best mechanical keyboard programming 2025`
**Result:** SUCCESS. Returned 5 Reddit results with full URLs containing post IDs
(e.g., `reddit.com/r/keyboards/comments/{postId}/...`). Post IDs extractable for Reddit MCP
tools. Already validated in Iteration 11; re-confirmed here.
**Conclusion:** Keep as documented.

## Changes Made

**researcher-prompt.md "Accessing Restricted Content" section:**

1. **Paywalled articles:** Demoted archive.ph (CAPTCHA-blocked), promoted Wayback Machine to
   primary with exact-timestamp URL format requirement
2. **Twitter/X:** Added `proxy: "stealth"` + `waitFor: 5000` requirement for xcancel.com
   (5 credits). Previous guidance didn't mention these were needed.
3. **Medium:** Replaced freedium.cfd with freedium-mirror.cfd, removed archive.ph fallback
4. **Bluesky:** Removed raw API endpoint. Documented that bsky.app is a JavaScript SPA
   requiring `waitFor: 5000` for Firecrawl scrape. No proxy needed (1 credit).
5. **Reddit/GitHub/Discourse/LinkedIn:** Unchanged (not retested, already validated)

## Credit Cost Summary

Total Firecrawl credits used: ~38
- archive.ph: 10 (2 attempts x 5 credits stealth/enhanced)
- Wayback Machine: 4 (4 attempts x 1 credit basic)
- xcancel.com: 16 (1 basic + 3 stealth attempts)
- freedium.cfd: 0 (DNS failure, no credit charged)
- freedium-mirror.cfd: 1 (basic proxy)
- Bluesky: 8 (API 5 auto-stealth + 3 bsky.app scrapes x 1 basic)
- firecrawl_search: ~0 (search credits separate)

# Iteration 13 -- Information Cascade Detection (Backlog #7)

## Type: Research Run (Empirical) + Engineering

## Question

"How do journalists, fact-checkers, and researchers detect information cascades -- cases where many
sources all trace back to one original, creating an illusion of independent corroboration?"

## Configuration

- **Question type:** Technical
- **Scope:** Broad (3 sonnet researchers, 2 rounds)
- **Research angles:**
  1. Journalism/fact-checking practitioner methods (Researcher A)
  2. Academic citation cascade analysis (Researcher B)
  3. Computational misinformation cascade detection at scale (Researcher C)

## Report

`~/.claude/research/information-cascade-detection/report.md`
~180 lines, 37 footnotes, ~50+ sources across researchers.

## Key Research Findings

**Three converging research traditions** all address cascade detection but have developed largely
independently:

1. **Journalism/fact-checking:** "Trace to source" methodology. First Draft's provenance pillar,
   GIJN's "don't rely on other media" rule, IFCN two-source minimum. No org publishes an explicit
   cascade detection checklist -- it's tacit professional knowledge. The ivermectin/Rolling Stone
   case is the paradigmatic cascade case study.

2. **Academic bibliometrics:** Greenberg 2009 (BMJ) is foundational -- identified citation bias,
   amplification, and invention across 242 papers. Woozle effect quantified by Letrud & Hernes
   2019 (76% of citing articles affirm debunked myths). Citogenesis (Wikipedia circular loops)
   well-documented. Chen et al. 2025 confirmed "telephone effect" computationally at 13M-pair
   scale.

3. **Computational social science:** Vosoughi et al. 2018 (Science) foundational -- false news 6x
   faster, distinct cascade topology. CrowdTangle shutdown (Aug 2024) left major monitoring gap.
   DisTrack and temporal graph approaches (TIDE-MARK) are current state-of-art. AI amplifies
   cascades via knowledge collapse and epistemic destabilization.

**Practitioner heuristics (Round 2):** Specific red flags distilled: identical phrasing/errors
propagated, tight temporal clustering, all sources citing same original, lack of local detail,
same stock photos. Source TYPE diversity (documents + people + data) matters more than source
COUNT. Intelligence analysis (Heuer ACH, ICD 203) has the most rigorous frameworks for source
independence evaluation.

**Observable signals (Round 2):** Content+structure+style similarity distinguishes cascade from
independent reporting (Bar et al. 2012). Temporal clustering is a primary signal -- one-quarter
of news stories reproduced within 4 minutes (Cage et al. 2025). Network topology differs
systematically (broadcast/star = cascade, bushy/multi-origin = independent).

## Dogfooding Observations

**What worked well:**
- All three Round 1 angles produced highly complementary material with minimal overlap
- Lead-dispatched follow-ups worked perfectly (8th consecutive success)
- Round 2 targeted the right gaps -- practitioner heuristics and observable signals filled the
  key gaps from Round 1
- Researcher B had no Round 2 task (academic angle was thoroughly covered in Round 1) -- idle
  researcher management worked correctly

**What went less well:**
- Researcher A did not write a feedback file before shutdown (2 of 3 wrote feedback)
- Researchers B and C both flagged Firecrawl returning oversized results for long academic
  articles (662KB for BMJ paper), with JSON format making partial reads difficult

**Feedback themes (from B and C):**
- Exa summaries were excellent for academic paper triage -- often sufficient without deep scraping
- Firecrawl PDF parser failed on one conference paper (MCP error)
- Suggestion: structured JSON extraction with targeted schema would be more token-efficient than
  full markdown for long academic papers
- Intelligence analysis angle (Heuer ACH, OSINT corroboration vs replication) was an
  underexplored rich connection

## Changes

### 1. Added "Cascade Check" section to researcher-prompt.md (lines ~250-290)

New section between Stopping Criteria and Handling Insufficient Evidence. Contains:
- 5-step cascade check procedure (trace citation chain, check copied phrasing, assess source
  type diversity, apply "how do they know?" test, watch for wire service cascades)
- Red flag list (temporal clustering, no local detail, single-origin collapse, same source type)
- Instruction to downgrade confidence and flag in CROSS-SOURCE ANALYSIS when detected

### 2. Added cascade check reference to Step 3 (Reflect)

Added seventh reflection question: "Cascade check (see below): Are my 'multiple sources'
actually independent?" -- points researchers to the new section.

## Backlog Updates

- **#7 (Information cascade detection):** RESOLVED. Research complete, heuristics engineered
  into researcher reflection step.
- **#13 (researcher-prompt.md length):** File now ~541 lines (up from ~497). Added ~44 lines.
  No evidence of instruction-following degradation across 12 prior iterations, but worth
  monitoring. Next empirical run should verify researchers find and follow the cascade check.
- **New observation:** Firecrawl returns oversized results for long academic HTML pages.
  Researchers suggest structured JSON extraction with schema. Not adding to backlog -- the Exa
  summary pattern is the recommended workaround and researchers are already following it.

## Post-Iteration Changes

Discussion after the research run led to two design changes applied to SKILL.md:

### 1. Flexible round counts (SKILL.md)

Replaced rigid per-scope round counts with ranges and convergence-based stopping:

| Scope         | Old | New |
| ------------- | --- | --- |
| Focused       | 1   | 1-2 |
| Broad         | 2   | 2-3 |
| Comprehensive | 2-3 | 3-4 |

Added early stopping language: rounds are heuristics, not targets. Stop when citation
convergence is reached. Removed hard phase-skip rules (e.g., "For Focused scope: skip to
Phase 8") and replaced with convergence checks at each triage point. Any scope can exit
early if findings have converged, and any scope can run additional rounds if gaps persist
within its budget.

### 2. Verification architecture rethink (Backlog #16 promoted)

Identified that the current verification model (reusing angle researchers in Round 3,
Comprehensive only) has two problems:

1. **Angle researchers are specialists** -- pulling them off their angle to do cross-cutting
   verification wastes their accumulated context and expertise.
2. **Verification doesn't need a round** -- it's a parallel task, not a sequential phase.

Promoted backlog #16 (dedicated verifier agents) to High Priority with full design:
- Spawn lightweight, single-purpose verifier agents on-demand after any round
- Verifiers run in parallel with angle researchers continuing their work
- Lead assigns specific claims to verify based on triage
- Available at any scope, not just Comprehensive
- Implementation deferred to a future iteration

# Synthesis Template: Consumer

Ranked recommendations with methodology.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Recommendations

### Top Pick: [Product]

[Why, with evidence from expert testing]

### Runner-Up: [Product]

[Why, differentiation from top pick]

### Budget Pick: [Product]

[Trade-offs at this price point]

### Best for {Use Case}: [Product]

[Specific use-case fit]

## Methodology

[Criteria used, sources consulted, testing methodology of source reviewers]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```

# Synthesis Template: Contested

Structured debate -- NEVER resolve with a verdict.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Perspectives

### Position A: [Label]

[Strongest case for this position, steelmanned]

### Position B: [Label]

[Strongest case for this position, steelmanned]

## Points of Agreement

[Where the sides converge]

## Key Disagreements

[Where they diverge and why -- identify factual vs. values-based disagreements]

## Evidence Quality by Position

[Assess the strength of evidence supporting each side -- this is NOT a verdict]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```

# Synthesis Template: Emerging/Frontier

State of play with instability warning.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Current State (as of {date})

[What is known now, with explicit date markers on key facts]

## Trajectory

_Note: The following is projection based on current trends, not established fact._

[Where things appear to be heading]

## Instability Warning

[What is most likely to change and why -- flag specific claims with short shelf life]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```

# Synthesis Template: Factual

Clear answer with evidence weight.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Verdict

[Direct answer to the question with confidence level]

## Supporting Evidence

### [Claim 1]

[Evidence with inline citations, weighted by source authority]

### [Claim 2]

[Continue same pattern]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```

# Synthesis Template: Opinion/Sentiment

Distribution mapping with representativeness assessment.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Distribution of Views

### Majority View (~X%)

[What most people think and why]

### Minority View (~X%)

[Dissenting perspective and reasoning]

### Outlier Positions

[Fringe views worth noting]

## Representativeness Assessment

[How representative is the source sample? What communities were/weren't covered?]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```

# Synthesis Template: Scientific/Health

Evidence pyramid with study quality assessment.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Evidence Summary

### Systematic Reviews & Meta-Analyses

[Findings from highest-quality evidence]

### Randomized Controlled Trials

[Key RCT findings]

### Observational Studies

[Cohort/case-control findings]

## Study Quality Notes

[Sample sizes, methodology concerns, funding disclosures]

## Practical Implications

_Note: The following reflects interpretation of the evidence, not sourced fact._

[What the evidence means in practice]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```

# Synthesis Template: Technical

Comparison with context and benchmarks.

```markdown
## Executive Summary

[2-3 sentences: key finding, confidence level, scope]

## Comparison Matrix

| Criterion   | Option A | Option B | Option C |
| ----------- | -------- | -------- | -------- |
| [criterion] | ...      | ...      | ...      |

## Detailed Analysis

### [Criterion 1]

[Evidence with benchmarks and citations]

### [Criterion 2]

[Continue same pattern]

## Recommendation

[Context-dependent recommendation with caveats about when each option wins]

## Confidence Assessment

[Use calibrated language from the Confidence table in SKILL.md]

- Almost certain (93-99%): [findings]
- Highly likely (80-92%): [findings]
- Likely (63-79%): [findings]
- Roughly even / Unlikely: [findings, if any]

## Limitations & Gaps

[What remains uncertain, missing, or contested]

## Sources

[^1]: Author/Org. "Title". Publication. URL

[^2]: ...
```