Abstract
AI Scientist systems capable of autonomous research hold the potential to dramatically accelerate scientific discovery. Yet, evaluating and comparing the quality of AI-generated papers remains an open challenge. We propose and implement a rigorous benchmarking protocol using an automated peer-review system that leverages frontier large language models (LLMs) to assess scientific papers across four core dimensions: originality, scientific rigor, clarity, and significance. We evaluate four leading AI Scientist frameworks—Sakana AI (v1 & v2), CycleResearcher, and Data-to-Paper—each run on a consistent set of 15 research proposals from a commercial autonomous AI scientist company (FARS), generating 60 papers that we assess alongside 15 FARS benchmark papers. Using three independent LLM reviewers (GPT-5.4, Gemini, and Claude), we find that FARS benchmark papers significantly outperform all competing frameworks, achieving mean scores of 2.14–2.47 on a 1–5 scale, compared to 1.00–1.87 for other systems. Notably, FARS scores are more than 2× higher than the next-best systems in Gemini and Claude evaluations. We observe strong agreement between Gemini and Claude (ρ = 0.907, p < 0.001), and both correlate extremely strongly with the synthesis score (ρ = 0.961, p < 0.001), validating the reliability of automated evaluation. However, GPT-5.4 exhibits weaker agreement (ρ ≈ 0.32), suggesting it evaluates papers using different criteria. These results establish the first quantitative benchmark for AI Scientist systems and demonstrate that multi-model LLM evaluation provides a scalable, consistent framework for assessing autonomous research quality.
1. Introduction
The emergence of AI Scientist systems—autonomous frameworks capable of generating, executing, and reporting research—marks a transformative shift in scientific discovery. By 2026, such systems have moved from theoretical proposals to practical tools, with commercial entities like FARS deploying them in real-world R&D pipelines. However, this progress introduces a critical bottleneck: how to objectively evaluate and compare the output quality of these systems. Traditional peer review, while valuable, is labor-intensive, subjective, and insufficiently scalable for the volume of AI-generated research. To address this gap, we present a benchmarking study that employs an automated multi-model review protocol to assess the quality of papers produced by four leading AI Scientist frameworks.
2. Methodology
2.1 AI Scientist Frameworks
We evaluated four prominent AI Scientist systems: Sakana AI (versions 1 and 2), CycleResearcher, and Data-to-Paper. Each system was tasked with generating papers from a consistent set of 15 research proposals provided by FARS, a commercial autonomous AI scientist company. This controlled setup ensured comparability across frameworks.
2.2 Automated Review Protocol
We designed an automated peer-review system using three frontier LLMs—GPT-5.4, Gemini, and Claude—to assess each paper on four dimensions: originality, scientific rigor, clarity, and significance. Reviewers scored papers on a 1–5 scale, and a synthesis score was computed by aggregating the dimension scores.
2.3 Dataset
In total, 60 papers were generated by the four frameworks (15 papers each) and evaluated alongside 15 FARS benchmark papers, yielding 75 papers assessed in this study.
3. Results
3.1 Performance Comparison
FARS benchmark papers significantly outperformed all competing frameworks, with mean scores ranging from 2.14 to 2.47 across the three reviewers. In contrast, other systems scored between 1.00 and 1.87. Notably, FARS scores were more than twice as high as the next-best frameworks in evaluations by Gemini and Claude.
3.2 Reviewer Agreement
We found strong agreement between Gemini and Claude (Spearman's ρ = 0.907, p < 0.001), indicating consistent evaluation criteria. Both reviewers also correlated highly with the overall synthesis score (ρ = 0.961, p < 0.001), validating the reliability of the automated evaluation process. However, GPT-5.4 showed weaker agreement with the others (ρ ≈ 0.32), suggesting that it may prioritize different attributes when assessing paper quality.
4. Discussion
Our findings establish a first-of-its-kind quantitative benchmark for AI Scientist systems. The strong inter-rater reliability between Gemini and Claude supports the feasibility of scalable, multi-model automated review. The divergence observed with GPT-5.4 highlights the importance of reviewer diversity in capturing a comprehensive view of research quality. Future work should explore extending the benchmark to broader domains, incorporating human-in-the-loop validation, and investigating the root causes of GPT-5.4's disparate scoring behavior.
5. Conclusion
This study demonstrates that multi-model LLM evaluation can serve as a reliable and scalable framework for assessing autonomous research quality. The significant performance gap between FARS and other systems underscores the need for continued advancement in AI Scientist frameworks. As autonomous research becomes more prevalent, standardized benchmarking protocols like ours will be essential for tracking progress and ensuring high scientific standards.
via ArXiv AI