Scholar Loop

An autonomous AI scientist: a multi-agent loop over literature, experiments, self-critique and write-up, with deterministic guards against reward-hacking and ha…

LLM Mart 4 views 289k listing impressions

read papers → find a gap → run real experiments → reflect → write & self-review

tests python license status


ScholarLoop runs the loop a PhD actually runs: it reads the literature and writes you a review, proposes novelty-checked ideas and lets you pick, runs real ML experiments on the chosen one, scores them against a frozen ground-truth metric, learns from its failures, and then writes it up the slow way — outline, key points, prose — with AI panels arguing each step and you signing off before the next. A deterministic harness keeps the agents honest and impossible to reward-hack.

Stage What it does
📚 Lit Review plans queries → pulls real papers (arXiv + OpenAlex) → themes · per-paper notes · open gaps → lit_review.md
💡 Ideator gaps + your engine's setting + what the ledger tried → N distinct, cited ideas, novelty-checked → ideas.md
🧑‍🔬 You read the review, push back; read the ideas, pick one (or push back) → handoff.json
🎯 Director (unattended runs) reads ledger + literature trends → sets the next direction & topic
🔭 Lit Scout pulls real papers from arXiv + OpenAlex, citation-ranked → structured, cited findings
💡 Reasoner constraints + literature + past lessons → the next experiment
🗳️ Debate Panel three personas vote — is this worth a GPU?
🪜 Funnel smoke → verify → full · a cheap screen kills most ideas
⚙️ Runner runs a real torch experiment, scored by a frozen metric
🪞 Reflector turns the outcome into a lesson in a decaying skill library
🚦 Advisor PROCEED · REFINE · PIVOT — steers the loop
✍️ Writing session outline → key points → prose, each argued by a Skeptic · Editor · Methodologist panel and approved by you; every number, citation and sentence checked in code
🧑‍⚖️ Reviewer panel three persona reviewers score the draft against a NeurIPS-style checklist → review.md
🗄️ Ledger + Skills the durable memory every step reads from

🧑‍🔬 Ideation first — a review you can read, ideas you can choose

The expensive part of research isn't running experiments, it's deciding which experiment deserves them. So the loop has a front half that stops and asks:

  topic ──▶ [1] literature review ──▶ lit_review.md   ◀── you: approve, or "also cover mixup"
                   │                                       (new queries · corpus grows · review rewritten)
                   ▼
            [2] candidate ideas ──▶ ideas.md          ◀── you: pick #2, or "be bolder"
                   │                                       (re-ideate on the same corpus)
                   ▼
            [3] handoff.json ──▶ Orchestrator.adopt() ──▶ smoke → verify → full …
export ANTHROPIC_API_KEY=sk-ant-...
python -m scholarloop.session --profile profiles/digits-mlp.yaml \
    --topic "regularization recipes for small MLPs" --out runs/s1       # interactive
#   (a)pprove · (r)efine <feedback> · (q)uit  >  r also cover data-side regularization such as mixup
#   (s)elect <n> · (r)efine <feedback> · (q)uit >  s 2
SCHOLARLOOP_HANDOFF=runs/s1/handoff.json SCHOLARLOOP_OUT=runs/s1 \
    python examples/run_to_paper.py                                      # run the chosen idea, then write it up

From the project's README.

Comments (0)

Sign in to join the conversation.

No comments yet.