read papers → find a gap → run real experiments → reflect → write & self-review
ScholarLoop runs the loop a PhD actually runs: it reads the literature and writes you a review, proposes novelty-checked ideas and lets you pick, runs real ML experiments on the chosen one, scores them against a frozen ground-truth metric, learns from its failures, and then writes it up the slow way — outline, key points, prose — with AI panels arguing each step and you signing off before the next. A deterministic harness keeps the agents honest and impossible to reward-hack.
| Stage | What it does |
|---|---|
| 📚 Lit Review | plans queries → pulls real papers (arXiv + OpenAlex) → themes · per-paper notes · open gaps → lit_review.md |
| 💡 Ideator | gaps + your engine's setting + what the ledger tried → N distinct, cited ideas, novelty-checked → ideas.md |
| 🧑🔬 You | read the review, push back; read the ideas, pick one (or push back) → handoff.json |
| 🎯 Director | (unattended runs) reads ledger + literature trends → sets the next direction & topic |
| 🔭 Lit Scout | pulls real papers from arXiv + OpenAlex, citation-ranked → structured, cited findings |
| 💡 Reasoner | constraints + literature + past lessons → the next experiment |
| 🗳️ Debate Panel | three personas vote — is this worth a GPU? |
| 🪜 Funnel | smoke → verify → full · a cheap screen kills most ideas |
| ⚙️ Runner | runs a real torch experiment, scored by a frozen metric |
| 🪞 Reflector | turns the outcome into a lesson in a decaying skill library |
| 🚦 Advisor | PROCEED · REFINE · PIVOT — steers the loop |
| ✍️ Writing session | outline → key points → prose, each argued by a Skeptic · Editor · Methodologist panel and approved by you; every number, citation and sentence checked in code |
| 🧑⚖️ Reviewer panel | three persona reviewers score the draft against a NeurIPS-style checklist → review.md |
| 🗄️ Ledger + Skills | the durable memory every step reads from |
🧑🔬 Ideation first — a review you can read, ideas you can choose
The expensive part of research isn't running experiments, it's deciding which experiment deserves them. So the loop has a front half that stops and asks:
topic ──▶ [1] literature review ──▶ lit_review.md ◀── you: approve, or "also cover mixup"
│ (new queries · corpus grows · review rewritten)
▼
[2] candidate ideas ──▶ ideas.md ◀── you: pick #2, or "be bolder"
│ (re-ideate on the same corpus)
▼
[3] handoff.json ──▶ Orchestrator.adopt() ──▶ smoke → verify → full …
export ANTHROPIC_API_KEY=sk-ant-...
python -m scholarloop.session --profile profiles/digits-mlp.yaml \
--topic "regularization recipes for small MLPs" --out runs/s1 # interactive
# (a)pprove · (r)efine <feedback> · (q)uit > r also cover data-side regularization such as mixup
# (s)elect <n> · (r)efine <feedback> · (q)uit > s 2
SCHOLARLOOP_HANDOFF=runs/s1/handoff.json SCHOLARLOOP_OUT=runs/s1 \
python examples/run_to_paper.py # run the chosen idea, then write it up
No comments yet.