arXiv:2607.28631v1 Announce Type: new
Abstract: AI Scientist systems capable of autonomous research have the potential to significantly accelerate scientific discovery. However, evaluating and comparing the quality of AI-generated papers remains an open challenge. We propose and implement a rigorous benchmarking protocol using an automated peer-review system that harnesses frontier large language models to assess scientific papers across four core dimensions: originality, scientific rigor, clarity, and significance. We evaluate four leading AI Scientist frameworks: textit{Sakana AI (v1 & v2)}, textit{CycleResearcher}, and textit{Data-to-Paper}. Each framework was run on a consistent set of 15 research proposals published by a commercial autonomous AI scientist company (FARS), generating 60 papers that we evaluate alongside 15 FARS benchmark papers. Using three independent LLM reviewers (GPT-5.4, Gemini, and Claude), we find that FARS benchmark papers significantly outperform all competing frameworks, achieving mean scores of 2.14–2.47 on a 1–5 scale compared to 1.00–1.87 for other systems. Notably, FARS scores are more than 2$times$ higher than the next-best systems on Gemini and Claude evaluations. We find strong agreement among Gemini and Claude ($rho$ = 0.907, $p
Abstract: AI Scientist systems capable of autonomous research have the potential to significantly accelerate scientific discovery. However, evaluating and comparing the quality of AI-generated papers remains an open challenge. We propose and implement a rigorous benchmarking protocol using an automated peer-review system that harnesses frontier large language models to assess scientific papers across four core dimensions: originality, scientific rigor, clarity, and significance. We evaluate four leading AI Scientist frameworks: textit{Sakana AI (v1 & v2)}, textit{CycleResearcher}, and textit{Data-to-Paper}. Each framework was run on a consistent set of 15 research proposals published by a commercial autonomous AI scientist company (FARS), generating 60 papers that we evaluate alongside 15 FARS benchmark papers. Using three independent LLM reviewers (GPT-5.4, Gemini, and Claude), we find that FARS benchmark papers significantly outperform all competing frameworks, achieving mean scores of 2.14–2.47 on a 1–5 scale compared to 1.00–1.87 for other systems. Notably, FARS scores are more than 2$times$ higher than the next-best systems on Gemini and Claude evaluations. We find strong agreement among Gemini and Claude ($rho$ = 0.907, $p