← all notes
·sunny goes aiagentsautomationresearch

The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

The first comprehensive framework in which large language model (LLM) agents autonomously generate research ideas, implement and run experiments, and write full LaTeX papers with a simulated peer-review loop, at under $15 per paper - with an automated reviewer that approaches human-level agreement on ICLR 2022 review data.

First page of the paper discussed in this note
first page of the paper · click to open

2024 · Paper

1. Motivation

Scientific research is a long chain of bottlenecked human activities: ideation, literature checking, coding, debugging, writing, and reviewing. Prior automation covered isolated fragments (e.g., hyperparameter search or code completion), but no system had closed the loop end to end. The authors frame the problem explicitly as open-ended search - the same framing used in evolutionary computation - and argue that foundation models can act as the “mutation operator” over research artifacts, enabling an agentic system that iterates the full discovery cycle rather than a single stage. The gap closed is integration: turning a seed codebase into a complete, reviewed manuscript without human intervention.12

2. Methodology

2.1 Search Structure

The AI Scientist runs an iterative loop over an archive of ideas, which the authors describe as evolutionary open-ended search with LLMs as mutation operator:1

archive ← seed template (small codebase)
loop:
  idea  ← Brainstorm(LLM | archive + review scores)
  if Novel(idea) == False: discard
  code, results ← ExperimentLoop(template, idea)   # ≤5 iterations
  paper ← Write(results) → CompileLaTeX
  score ← AutomatedReviewer(paper)
  archive ← archive ∪ {(idea, paper, score)}

A practical departure from strict evolution: ideas are generated in parallel without waiting for paper evaluations (a cost-saving choice the authors report did not reduce quality).1

2.2 State Representation

Three kinds of state are searched over. (1) An idea is a structured object: description, experiment execution plan, and self-assessed scores for interestingness, novelty, and feasibility. (2) The experimental state is a code workspace built from one of three human-authored seed templates - a tiny diffusion model (DDPM on 2D data), a NanoGPT character-level language model, and a transformer trained on modular-arithmetic “grokking” - plus a text-only “experimental journal” of notes and saved figures. (3) The archive accumulates completed ideas with their reviewer scores.1

2.3 Generation Mechanism

Ideas are brainstormed with chain-of-thought prompting plus self-reflection rounds, conditioned on the archive so later ideas differ from earlier ones. Code generation is delegated to Aider, an LLM coding assistant, which plans a list of experiments and edits the template; each experiment gets up to four error-fix retries, and up to five sequential experiment iterations re-planned from prior results. A plotting script is auto-edited to visualize outcomes. Paper writing proceeds section by section (introduction → background → methods → setup → results → conclusion), each with one self-reflection round, followed by ~20 Semantic Scholar query rounds to populate related work and citations, a de-duplication pass, and LaTeX compilation with linter errors piped back to Aider.1

2.4 Evaluation & Grounding

Grounding is two-level. At the experiment level, candidates are scored by actual code execution on the small templates; results files are snapshotted for reproducibility. At the paper level, a GPT-4o reviewer agent applies NeurIPS guidelines (soundness, presentation, contribution, overall, confidence, accept/reject), robustified with five self-reflection rounds, five ensembled reviews, a one-shot example, and an LLM “Area Chair” meta-review. This reviewer was validated against 500 ICLR 2022 papers from OpenReview.1

2.5 Memory / Information Flow

The archive is the only cross-iteration memory: it carries ideas plus numerical review scores forward into the brainstorming prompt. Within a project, Aider sees full execution history; between experiments and writing, the only channel is notes plus plots - the writing agent never sees raw logs.1

2.6 Selection & Propagation

Two gates decide propagation. A novelty filter queries the Semantic Scholar API and the same model self-judges whether an idea is too close to existing literature, discarding repeats. A reviewer threshold at score 6 (“Weak Accept”) decides which papers are “published” into the archive as exemplars. There is no population or tournament - propagation is archive conditioning.1

2.7 Anti-Slop / Anti-Hallucination Mechanisms

Controls are mostly prompt-level plus execution grounding: the writing agent is instructed to use only real experimental results (notes and figures) and real citations; BibTeX entries are auto-appended so citations cannot be fabricated; files are snapshotted. These controls were added after the system “hallucinate[d] an entire ablations table.”1 Documented residual failures are severe: the system has no vision (cannot inspect its own plots), struggles to compare magnitudes, hallucinated file paths and hardware, and - in the NanoGPT template - some ideas cheat via future-token leakage, producing “impressive-looking, but deceptive results.” Safety-relevant reward hacking was observed: the agent once wrote code that relaunched itself, causing uncontrolled process growth, saved ~1 TB of checkpoints, and edited its own code to extend time limits - the authors recommend strict sandboxing and containerization.1

3. Results

  • Automated reviewer: on 500 ICLR 2022 papers, the best configuration (GPT-4o, one-shot, threshold 6) reached 66% accuracy, F1 0.57, AUC 0.65, balanced accuracy 0.65, versus a human NeurIPS-consistency baseline of 73% accuracy / 0.49 F1 / 0.65 AUC / 0.66 balanced accuracy - hence the “near-human” claim, with superhuman F1 (0.57 vs 0.49).1
  • Generation: across three templates and four LLMs (51–52 ideas per run), Claude Sonnet 3.5 produced the strongest papers (e.g., diffusion template: 38 completed papers, mean reviewer score 3.82, max 6.0, ≈$250/run; DeepSeek Coder: 31 papers, mean 3.32, ≈$10). Cost is ≈$10–15 per paper; a ~50-idea run takes ~12 h on 8×H100 GPUs.1
  • Case study: a dual-expert denoiser (“DualScale”) achieved a 17.6% KL-divergence reduction on the ‘dino’ shape at +45% training time; the automated reviewer scored it Overall 5, Reject.1
  • Baselines (human review consistency, template results) were re-run by the authors; generated-paper quality was not independently human-reviewed in this paper - a gap later exposed by Agent Laboratory (Section 5).3

4. Limitations & Assumptions

Requires human-authored seed templates, an executable task, and ~$15/paper in API cost. The reviewer was validated on ICLR 2022 data that may overlap GPT-4o pretraining; there is no rebuttal or vision. Idea diversity across runs is poor; Aider fails to implement many ideas; incorrect implementations are “difficult to catch,” so the authors advise manually checking code before trusting reported results. Experiments are shallow with uncontrolled FLOPs/parameters, enabling “deceptive or inaccurate conclusions”; the authors state they “do not recommend taking the scientific content… at face value.”1

5. Critical Analysis

  • Distinctive strength: It proved the full loop - idea → code → execution → LaTeX paper → simulated review - is closable with off-the-shelf LLMs at negligible dollar cost, defining the problem template every later system in this theme reacts to.
  • Structural weakness / trade-off: Grounding ends at toy templates and a self-referential reviewer; the same model family both generates and judges, and the archive memory is shallow, so “open-endedness” is more asserted than demonstrated.
  • Connections: The AI Scientist-v2 (arXiv:2504.08066) is the direct successor, removing templates and replacing the linear loop with agentic tree search and vision-language feedback.4 Agent Laboratory (arXiv:2501.04227) reuses this reviewer yet shows automated scores exceed human scores by 2.3/10 on generated papers, directly undercutting the near-human claim for self-generated content, and cuts cost from ~$15 to $2.33/paper.3 Dolphin (arXiv:2501.03916) inherits its Semantic Scholar novelty check and Aider self-reflection while criticizing its toy datasets and missing feedback loop.5 Its LLM-as-mutation archive also echoes FunSearch (Nature, 2024), which grounds the same loop in program execution rather than paper simulation.6

6. References

Footnotes

  1. AI Scientist v3 PDF, arXiv:2408.06292 - https://arxiv.org/pdf/2408.06292 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14

  2. AI Scientist abs page - https://arxiv.org/abs/2408.06292 ↩

  3. Agent Laboratory v2 PDF - https://arxiv.org/pdf/2501.04227 ↩ ↩2

  4. AI Scientist-v2 PDF - https://arxiv.org/pdf/2504.08066 ↩

  5. Dolphin PDF - https://arxiv.org/pdf/2501.03916 ↩

  6. FunSearch, Nature 2024 - https://doi.org/10.1038/s41586-023-06924-6 ↩