Before starting Phase 1, check whether aggregation is actually needed:
\n| Situation | \nAction | \n
|---|---|
workspace/inputs/idea.md and workspace/inputs/experimental_log.md both exist and are non-empty | \nSkip this skill entirely. Proceed directly to paper-orchestra. | \n
| Either file is missing or empty, and the user provided a directory path | \nRun this skill with that directory as --search-roots. | \n
| Either file is missing or empty, and no directory was provided | \nScan cwd and ~ by default; show the discovery summary to the user before continuing. | \n
| The inputs exist but look thin (e.g. idea.md has < 5 lines, no numeric data in experimental_log.md) | \nAsk the user whether to supplement with aggregation or proceed as-is. | \n
The skill is intentionally a pre-pass — it is cheap to skip and should only run when the structured inputs don't already exist.
\nA pre-processing skill for PaperOrchestra (arXiv:2604.05018). Reads scattered\nexperimentation artifacts from AI coding-agent cache directories and synthesizes\nthem into the structured (I, E) input pair the PaperOrchestra pipeline expects.
[.claude/] [.cursor/] [.antigravity/] [.openclaw/]\n │ │ │ │\n └────────────┴──────────────┴───────────────┘\n │\n Phase 1: Discovery\n (discover_logs.py)\n │\n discovered_logs.json\n │\n Phase 2: Extraction\n (LLM call per log batch)\n │\n raw_experiments.json\n │\n Phase 3: Synthesis\n (LLM call — consolidate)\n │\n synthesis.json\n │\n Phase 4: Formatting\n (format_po_inputs.py)\n │\n ┌────────────┴────────────┐\n workspace/inputs/ workspace/ara/\n idea.md aggregation_report.md\n experimental_log.md discovered_logs.json\n raw_experiments.json\n synthesis.json\nThe output drops directly into workspace/inputs/ so the user can immediately\nrun paper-orchestra on the same workspace.
| Parameter | \nRequired | \nDefault | \nDescription | \n
|---|---|---|---|
--search-roots | \nno | \ncwd, ~ | \nComma-separated directories to scan for agent caches | \n
--agents | \nno | \nall | \nComma-separated subset: claude,cursor,antigravity,openclaw | \n
--workspace | \nno | \n./workspace | \nPaperOrchestra workspace root | \n
--depth | \nno | \n4 | \nMax directory scan depth (prevents runaway scans on large home dirs) | \n
--since | \nno | \nnone | \nOnly include logs modified after this date (ISO 8601: 2025-01-01) | \n
The user specifies these when invoking the skill, or you may ask them for\n--search-roots if the current directory has no detectable agent caches.
Run the discovery script to catalog every relevant log file:
\npython skills/agent-research-aggregator/scripts/discover_logs.py \\\n --search-roots <roots> \\\n --agents <agents> \\\n --depth <depth> \\\n --since <since> \\\n --out workspace/ara/discovered_logs.json\nThe script exits with code 2 when no --project filter is set (this is\nexpected on the first run). It prints a \"Projects found\" list to stdout —\nshow it to the user immediately.
If no logs are found at all: stop and ask the user to specify\n--search-roots or point you at a directory that contains agent cache folders.
A paper can only be written from a single project. You must ask the user\nwhich project to use before any LLM processing begins.
\nProjects found:\n [1] /home/alice/projects/my-rl-experiment (42 files)\n [2] /home/alice/projects/llm-eval-suite (17 files)\n [3] /home/alice/projects/old-demo (3 files)\npython skills/agent-research-aggregator/scripts/discover_logs.py \\\n --search-roots <roots> \\\n --agents <agents> \\\n --depth <depth> \\\n --since <since> \\\n --project \"<chosen project path>\" \\\n --out workspace/ara/discovered_logs.json\nThis overwrites discovered_logs.json so only the selected project's files\nremain. The script exits 0 on success.
If the discovery finds only one project: skip the question and inform the\nuser: \"Only one project found: <path>. Using it for the paper.\" — then\nre-run with --project automatically.
If the discovery summary shows irrelevant files after filtering: ask the\nuser whether to include or exclude them before continuing to Phase 2. Err on\nthe side of inclusion — the extraction prompt is conservative.
\nProcess discovered logs in batches (group by agent type; keep batches under\n~50 KB of raw text to stay within context limits):
\nFor each batch:
\n--list output tells you\nwhich file paths to read).references/extraction-prompt.md as\nyour system message.workspace/ara/raw_experiments.json.After all batches:
\npython skills/agent-research-aggregator/scripts/extract_experiments.py \\\n --discovered workspace/ara/discovered_logs.json \\\n --out workspace/ara/raw_experiments.json \\\n --validate-only\nRun this in --validate-only mode to check the combined JSON is well-formed\nand meets the minimum schema (experiments array non-empty, each entry has\nhypothesis or method or results). Fix any malformed entries before Phase 3.
Consolidate possibly-redundant experiment records from multiple agent caches into\na single coherent research narrative. This is ONE LLM call.
\nSystem message: Use references/synthesis-prompt.md verbatim.
User message:
\n<raw_experiments>\n{contents of workspace/ara/raw_experiments.json}\n</raw_experiments>\nThe LLM must return a synthesis.json with keys:
research_question — the overarching question being investigatedhypothesis — the core proposed solution / claimmethod_summary — how the approach works (concise, no data leakage)key_contributions — 2–5 bullet stringsexperimental_setup — datasets, metrics, baselines, implementation notesresults_tables — array of {title, headers[], rows[]} markdown-table objectsqualitative_observations — free-form text blocks (what worked, what didn't,\nfailure modes, ablation insights)iteration_history — ordered list of {iteration_id, change_description, outcome} entries if multiple iterations are detectedopen_questions — questions that remain unanswered in the logsSave to workspace/ara/synthesis.json.
\n\nNote: By this point, the user has already selected a single project in\nPhase 1.5. The synthesis should represent one coherent research thread. If\nthe LLM still surfaces multiple disconnected research questions, flag this\nas a data quality warning in the audit report (Phase 5) but do not re-ask\nfor project selection — that decision was made earlier.
\n
Convert synthesis.json into PaperOrchestra input files:
python skills/agent-research-aggregator/scripts/format_po_inputs.py \\\n --synthesis workspace/ara/synthesis.json \\\n --out workspace/inputs/\nThis generates two files:
\nworkspace/inputs/idea.md (Sparse variant)Follows the PaperOrchestra Sparse Idea format (arXiv:2604.05018, §3.1):
\n# [Synthesized Research Title]\n\n## Problem\n<2–4 sentence problem statement derived from research_question>\n\n## Hypothesis\n<hypothesis from synthesis>\n\n## Method\n<method_summary from synthesis>\n\n## Key Contributions\n<key_contributions as bullet list>\n\n## Open Questions\n<open_questions, if any>\nworkspace/inputs/experimental_log.mdFollows the PaperOrchestra Experimental Log format (App. D.3):
\n## 1. Experimental Setup\n<experimental_setup from synthesis, formatted as prose + sub-bullets>\n\n## 2. Raw Numeric Data\n<results_tables converted to GitHub-Flavored Markdown tables>\n\n## 3. Qualitative Observations\n<qualitative_observations from synthesis>\n\n### Iteration History\n<iteration_history as an ordered narrative, if present>\nAfter running the script, review both files with the user:
\nworkspace/inputs/idea.md aloud and ask: \"Does this accurately capture\nyour research question and method?\"workspace/inputs/experimental_log.md and ask:\n\"Are these the correct metrics and baselines?\"Revise based on feedback before proceeding to PaperOrchestra.
\npython skills/agent-research-aggregator/scripts/format_po_inputs.py \\\n --synthesis workspace/ara/synthesis.json \\\n --out workspace/inputs/ \\\n --report workspace/ara/aggregation_report.md\nThe --report flag makes the script also write aggregation_report.md, which\ncontains:
Show the report to the user. If the data quality section lists warnings, discuss\nthem before running paper-orchestra — garbage in, garbage out.
\nOnce the user has confirmed idea.md and experimental_log.md, the workspace\nis ready for the paper-orchestra pipeline. You still need:
| File | \nStatus | \nAction | \n
|---|---|---|
workspace/inputs/idea.md | \n✓ generated | \nuser review recommended | \n
workspace/inputs/experimental_log.md | \n✓ generated | \nuser review recommended | \n
workspace/inputs/template.tex | \nMISSING | \nask user to provide their conference LaTeX template | \n
workspace/inputs/conference_guidelines.md | \nMISSING | \nask user to provide (page limit, deadline, formatting rules) | \n
Tell the user exactly which two files are still needed, then offer to run\npaper-orchestra once they supply them.
| Situation | \nAction | \n
|---|---|
| Cache directory does not exist | \nSkip silently; note in report | \n
| File is binary or non-text | \nSkip; note in report | \n
| File > 200 KB | \nTruncate at 200 KB; note in report with path | \n
| LLM extraction returns malformed JSON | \nRe-prompt once with the parse error appended; if still malformed, log the batch as status: failed and continue | \n
Synthesis returns > 1 research_question | \nLog as data quality warning in audit report; do not re-ask for project (was selected in Phase 1.5) | \n
results_tables is empty after synthesis | \nWarn the user — PaperOrchestra's section-writing agent needs numeric data | \n
.claude/, .cursor/, .antigravity/, .openclaw/.idea.md or experimental_log.md. The extraction prompt instructs the LLM to strip PII; double-check before handoff.[UNVERIFIED] in the table rather than silently including it.# Phase 1: discover all projects (exits with code 2 — project selection required)\npython skills/agent-research-aggregator/scripts/discover_logs.py \\\n --search-roots . ~ --out workspace/ara/discovered_logs.json\n\n# Phase 1.5: re-run with chosen project (exits 0)\npython skills/agent-research-aggregator/scripts/discover_logs.py \\\n --search-roots . ~ \\\n --project \"/home/user/projects/my-chosen-project\" \\\n --out workspace/ara/discovered_logs.json\n\n# ... (Phase 2: LLM extraction calls, see above) ...\n\npython skills/agent-research-aggregator/scripts/extract_experiments.py \\\n --discovered workspace/ara/discovered_logs.json \\\n --out workspace/ara/raw_experiments.json --validate-only\n\n# ... (Phase 3: LLM synthesis call, see above) ...\n\npython skills/agent-research-aggregator/scripts/format_po_inputs.py \\\n --synthesis workspace/ara/synthesis.json \\\n --out workspace/inputs/ \\\n --report workspace/ara/aggregation_report.md\n