https://sunnygoes.com/notes Multiagent AI • Robotics • Security https://sunnygoes.com/notes/2026-07-27-20-evaluating-agentic-optimization-large-codebases/ https://sunnygoes.com/notes/2026-07-27-20-evaluating-agentic-optimization-large-codebases/ Fri, 17 Jul 2026 00:00:00 GMT FormulaCode is a live, repository-scale benchmark that evaluates agentic performance engineering against human-expert patches on 957 real bottlenecks mined from 245,477 PRs across 70 ASV-instrumented scientific-Python repositories, using correctness rollback, statistical significance testing, and an expert-relative Advantage metric with a built-in contamination probe. https://sunnygoes.com/notes/2026-07-27-05-mlevolve/ https://sunnygoes.com/notes/2026-07-27-05-mlevolve/ Mon, 01 Jun 2026 00:00:00 GMT A self-evolving multi-agent framework that repairs three failure modes of MLE agents - inter-branch information isolation, memoryless search, and one-shot generation - via Progressive Monte Carlo Graph Search, Retrospective Memory, and Hierarchical Planning with Adaptive Code Generation, reaching a 65.3% medal rate on MLE-Bench at half the standard budget (12 h vs. 24 h) and the best result on 11 of 15 AlphaEvolve math tasks. https://sunnygoes.com/notes/2026-07-27-17-mars-mars-plus/ https://sunnygoes.com/notes/2026-07-27-17-mars-mars-plus/ Tue, 19 May 2026 00:00:00 GMT MARS treats automated AI research as cost-constrained search over a space of whole software repositories, combining budget-aware MCTS with an efficiency-shaped reward, a Design-Decompose-Implement modular construction pipeline, and a comparative reflective memory whose lessons are distilled from diffs against the previous best solution. https://sunnygoes.com/notes/2026-07-27-08-gigaevo-shinkaevolve-thetaevolve/ https://sunnygoes.com/notes/2026-07-27-08-gigaevo-shinkaevolve-thetaevolve/ Mon, 17 Nov 2025 00:00:00 GMT Three independent open-source derivatives show that AlphaEvolve-style program evolution can be made reproducible and radically more sample-efficient - GigaEvo by engineering the unspecified infrastructure, ShinkaEvolve by principled parent/LLM selection (new circle-packing SOTA in 150 samples), and ThetaEvolve by batch generation plus test-time RL (new best-known bounds from a single 8B open model). https://sunnygoes.com/notes/2026-07-27-18-fm-agent/ https://sunnygoes.com/notes/2026-07-27-18-fm-agent/ Thu, 30 Oct 2025 00:00:00 GMT FM Agent combines LLM-driven mutation and crossover with a multi-island evolutionary architecture, cold-start population generation, domain-specific evaluators, and a Ray-based asynchronous infrastructure, achieving state-of-the-art results across MLE-Bench, ALE-Bench, KernelBench, and classical mathematics problems. https://sunnygoes.com/notes/2026-07-27-01-from-ai-for-science-to-agentic-science/ https://sunnygoes.com/notes/2026-07-27-01-from-ai-for-science-to-agentic-science/ Mon, 20 Oct 2025 00:00:00 GMT The survey argues that "Agentic Science" - large language model (LLM) agents autonomously running the full hypothesis–experiment–analysis–refinement cycle - constitutes a distinct stage of AI for Science, and unifies the field through a three-layer framework of five foundational capabilities, four core processes, and four application domains. https://sunnygoes.com/notes/2026-07-27-02-deep-research-survey-autonomous-research-agents/ https://sunnygoes.com/notes/2026-07-27-02-deep-research-survey-autonomous-research-agents/ Mon, 18 Aug 2025 00:00:00 GMT The survey defines "deep research" as a paradigm beyond retrieval-augmented generation (RAG), in which agents iteratively plan, retrieve, and synthesize web-grounded analytical reports, and organizes the field by a capability-centric four-stage pipeline - planning, question developing, web exploration, report generation - rather than by whole-system enumeration. https://sunnygoes.com/notes/2026-07-27-16-ml-master/ https://sunnygoes.com/notes/2026-07-27-16-ml-master/ Thu, 19 Jun 2025 00:00:00 GMT ML-Master integrates parallelized UCT-MCTS exploration with a reasoning model fed a selectively scoped parent-and-sibling memory (1.0), and ML-Master 2.0 extends this to day-scale autonomy via Hierarchical Cognitive Caching (HCC) that differentiates experience → knowledge → wisdom across tasks. https://sunnygoes.com/notes/2026-07-27-04-alphaevolve/ https://sunnygoes.com/notes/2026-07-27-04-alphaevolve/ Mon, 16 Jun 2025 00:00:00 GMT An evolutionary coding agent that orchestrates a Gemini LLM ensemble to make grounded, evaluator-verified code changes, thereby discovering provably correct algorithms - including the first improvement in 56 years over Strassen for 4×4 complex matrix multiplication (48 scalar multiplications) - and deployed optimizations in Google's production infrastructure. https://sunnygoes.com/notes/2026-07-27-07-openevolve/ https://sunnygoes.com/notes/2026-07-27-07-openevolve/ Sun, 01 Jun 2025 00:00:00 GMT An open-source reimplementation and extension of AlphaEvolve - an asynchronous pipeline (prompt sampler → LLM ensemble → evaluator pool → program database) that evolves entire code files in any language under MAP-Elites quality-diversity plus island migration - claiming "2–3× speedups on real hardware" and "state-of-the-art circle packing" (self-reported). https://sunnygoes.com/notes/2026-07-27-19-rd-agent/ https://sunnygoes.com/notes/2026-07-27-19-rd-agent/ Tue, 20 May 2025 00:00:00 GMT R&D-Agent splits machine learning engineering between a Researcher agent that converts performance feedback into ideas and a Developer agent that converts error feedback into code, runs multiple heterogeneous exploration traces in parallel, and fuses their partial results into composite solutions; v2 generalizes this into a two-phase, six-component formalization of the whole MLE workflow. https://sunnygoes.com/notes/2026-07-27-10-the-ai-scientist-v2/ https://sunnygoes.com/notes/2026-07-27-10-the-ai-scientist-v2/ Tue, 01 Apr 2025 00:00:00 GMT By removing human-authored code templates and adding a four-stage experiment manager with parallelized agentic tree search and vision-language model (VLM) feedback, The AI Scientist-v2 produced the first entirely AI-generated manuscript to pass genuine blind peer review (average score 6.33/10) at an ICLR 2025 workshop. https://sunnygoes.com/notes/2026-07-27-03-agentic-ai-for-scientific-discovery-survey/ https://sunnygoes.com/notes/2026-07-27-03-agentic-ai-for-scientific-discovery-survey/ Wed, 12 Mar 2025 00:00:00 GMT This 13-page survey argues that agentic AI - LLM systems capable of reasoning, planning, and autonomous decision-making - is transforming every stage of scientific research, but that the field's near-term value lies in augmenting rather than replacing human researchers, with literature review empirically the weakest and most failure-prone stage of current autonomous pipelines. https://sunnygoes.com/notes/2026-07-27-14-agentrxiv/ https://sunnygoes.com/notes/2026-07-27-14-agentrxiv/ Sat, 01 Mar 2025 00:00:00 GMT A centralized preprint server that lets autonomous agent laboratories upload and retrieve each other's research papers, enabling cumulative, collaborative progress - agents with access to prior agent-generated research improve 11.4% (relative) over baseline on MATH-500 while isolated agents plateau, and three parallel laboratories reach 13.7%. https://sunnygoes.com/notes/2026-07-27-24-llm-sr/ https://sunnygoes.com/notes/2026-07-27-24-llm-sr/ Sat, 01 Mar 2025 00:00:00 GMT Symbolic regression becomes far more sample-efficient when an LLM proposes executable equation skeletons conditioned on a natural-language problem specification while numerical optimizers fit the placeholder parameters, with a FunSearch-style islands experience buffer maintaining diverse high-scoring hypotheses. https://sunnygoes.com/notes/2026-07-27-15-aide-automated-mle-agent/ https://sunnygoes.com/notes/2026-07-27-15-aide-automated-mle-agent/ Tue, 18 Feb 2025 00:00:00 GMT AIDE frames machine learning engineering (MLE) as a code optimization problem over Python scripts and operationalizes trial-and-error as a greedy best-first tree search whose every node is grounded in an executed validation metric. https://sunnygoes.com/notes/2026-07-27-12-ai-co-scientist/ https://sunnygoes.com/notes/2026-07-27-12-ai-co-scientist/ Sat, 01 Feb 2025 00:00:00 GMT A Gemini-2.0 multi-agent system that generates, debates, and evolves natural-language scientific hypotheses under an Elo-ranked tournament with literature-grounded verification, outperforming state-of-the-art reasoning models and expert "best guesses" on expert-curated goals - and, uniquely among autonomous research systems, having three of its outputs (acute myeloid leukemia drug repurposing, liver-fibrosis targets, and an antimicrobial-resistance mechanism) validated in wet-lab experiments. https://sunnygoes.com/notes/2026-07-27-11-dolphin-closed-loop-auto-research/ https://sunnygoes.com/notes/2026-07-27-11-dolphin-closed-loop-auto-research/ Wed, 01 Jan 2025 00:00:00 GMT Dolphin closes the research loop that The AI Scientist left open - feeding experimental outcomes back into idea generation so that idea quality improves across cycles - and, evaluated on standard benchmarks rather than toy templates, produces methods comparable to state-of-the-art on tasks such as 3D point-cloud classification. https://sunnygoes.com/notes/2026-07-27-13-agent-laboratory/ https://sunnygoes.com/notes/2026-07-27-13-agent-laboratory/ Wed, 01 Jan 2025 00:00:00 GMT A pipeline of role-specialized LLM agents (PhD, Postdoc, ML Engineer, Software Engineer, Professor) plus two solver tools (mle-solver, paper-solver) turns a human-provided research idea into a literature review, experiments, and a LaTeX report for as little as $2.33 per paper - an 84% cost decrease versus prior autonomous research methods - with optional human "co-pilot" checkpoints that measurably improve output quality. https://sunnygoes.com/notes/2026-07-27-09-the-ai-scientist/ https://sunnygoes.com/notes/2026-07-27-09-the-ai-scientist/ Thu, 01 Aug 2024 00:00:00 GMT The first comprehensive framework in which large language model (LLM) agents autonomously generate research ideas, implement and run experiments, and write full LaTeX papers with a simulated peer-review loop, at under $15 per paper - with an automated reviewer that approaches human-level agreement on ICLR 2022 review data. https://sunnygoes.com/notes/2026-07-27-23-reevo/ https://sunnygoes.com/notes/2026-07-27-23-reevo/ Thu, 01 Feb 2024 00:00:00 GMT Reflective Evolution integrates LLM-generated "verbal gradients" - short-term comparative reflections between parent heuristics and long-term distilled design knowledge - into genetic programming over open-ended heuristic code, yielding state-of-the-art combinatorial-optimization heuristics within 100 evaluated candidates. https://sunnygoes.com/notes/2026-07-27-06-funsearch/ https://sunnygoes.com/notes/2026-07-27-06-funsearch/ Thu, 14 Dec 2023 00:00:00 GMT Pairing a frozen pretrained code LLM with a deterministic evaluator in an island-based evolutionary loop ("FunSearch" = searching in function space) yields new verifiable scientific discoveries - cap sets beating 20-year-best constructions and online bin-packing heuristics beating first-fit and best-fit - while the evaluator guards against LLM confabulation. https://sunnygoes.com/notes/2026-07-27-21-reflexion/ https://sunnygoes.com/notes/2026-07-27-21-reflexion/ Sun, 01 Oct 2023 00:00:00 GMT Language agents can be improved across trials without any weight updates by converting environment feedback into natural-language self-reflections stored in a bounded episodic memory, which acts as a "semantic gradient" conditioning subsequent attempts. https://sunnygoes.com/notes/2026-07-27-22-promptbreeder/ https://sunnygoes.com/notes/2026-07-27-22-promptbreeder/ Fri, 01 Sep 2023 00:00:00 GMT An evolutionary algorithm in which an LLM mutates both a population of task-prompts and the mutation-prompts that generate those mutations - a self-referential loop that sustains diversity and escapes the diminishing returns of one-shot automatic prompt engineering. https://sunnygoes.com/notes/2026-07-27-25-map-elites/ https://sunnygoes.com/notes/2026-07-27-25-map-elites/ Wed, 01 Apr 2015 00:00:00 GMT MAP-Elites maintains a discretized archive over user-chosen feature dimensions and keeps the best-performing solution found so far in each cell, thereby illuminating the fitness potential of the whole feature space - and, by exploring more, often finding a better overall solution than optimization-focused search.