AI-Assisted Pre-Review of Open-Source Software Submissions: An Experience Report from BOSC 2026
Tazro Ohta, Nomi L. Harris, Seth Carbon
Abstract
Most conferences rely on peer review to evaluate submissions. However, as generative AI tools make it easier than ever to prepare submission materials, many conferences are experiencing a surge in submissions that can overwhelm volunteer reviewers. To address this, we explored whether generative AI could assist by pre-reviewing abstracts against predefined criteria. The Bioinformatics Open Source Conference (BOSC) was an ideal testbed, as it already had a detailed rubric for evaluating submissions on multiple dimensions, including openness (public availability of code or associated content), valid open-source licensing, and "runnability" (the ease of downloading, building, and running the project—a key indicator of reusability).
For BOSC 2026, we developed two key tools: bosc-pre-review, an agentic skill that assessed six review criteria, and Runabilly, which automatically builds and tests each project in a disposable Docker container for safety. Importantly, the AI only gathered evidence for reviewers; all decisions regarding acceptance remained in human hands. After the review period, we surveyed reviewers to gauge the usefulness of the pre-review. Most respondents found it valuable, but preferred to verify the AI's conclusions against their own judgment rather than accept them unquestioningly.
1. Introduction
The peer-review process is a cornerstone of academic and scientific conferences, ensuring quality and relevance of presented work. Yet, the rise of generative AI has democratized content creation, leading to a dramatic increase in submission volumes. While this reflects broader accessibility, it places a significant burden on reviewers, who are often volunteers with limited time. In 2026, this challenge has become even more pronounced, with conferences across disciplines reporting unprecedented submission numbers.
At BOSC, we recognized an opportunity to leverage generative AI not as a replacement for human judgment, but as a tool to augment efficiency. Our goal was to automate the initial screening of abstracts, allowing reviewers to focus on nuanced evaluation and decision-making.
2. Background and Motivation
BOSC has a well-established review process based on a detailed rubric. This rubric includes criteria such as:
- Openness: Whether the project's code or content is publicly available.
- Valid open-source license: Ensuring submissions comply with open-source definitions.
- Runnability: The ease with which a project can be downloaded, built, and run, which is crucial for reusability.
In previous years, reviewers manually assessed each submission against these criteria, which was time-consuming and often led to reviewer fatigue. The surge in submissions for BOSC 2026 highlighted the need for a more scalable approach.
3. System Design and Implementation
We developed two complementary components:
3.1 bosc-pre-review
This agentic skill was designed to automatically evaluate submissions against six review criteria. It used natural language processing and machine learning to analyze abstracts and identify evidence relevant to each criterion. The output was a structured summary for each submission, highlighting key points and potential issues.
3.2 Runabilly
To assess runnability, we implemented Runabilly, a tool that builds and tests each project in a disposable Docker container. This approach ensures safety by isolating the project from the host system. Runabilly automatically downloads the project, builds it, and runs basic tests to verify functionality, providing a reproducibility score.
Both tools were integrated into the BOSC review workflow, providing reviewers with pre-computed evidence that they could verify and interpret.
4. Evaluation and Results
After the review period, we conducted a survey among reviewers to assess the utility of the pre-review system. The survey focused on usability, accuracy, and trust. Key findings included:
- Positive reception: The majority of respondents found the pre-review useful, citing time savings and improved efficiency.
- Preference for human oversight: Reviewers preferred to cross-check the AI's conclusions against their own analysis, indicating a desire for transparency and control.
- Trust but verify: While AI was helpful in gathering evidence, reviewers did not want to rely on it blindly, emphasizing the importance of human-in-the-loop systems.
5. Discussion
The BOSC 2026 experiment demonstrates that generative AI can effectively assist in the pre-review of open-source software submissions, reducing workload while maintaining quality. However, the success of such systems depends on clear human oversight. The "trust but verify" approach adopted by reviewers aligns with broader principles of responsible AI deployment.
Future work could explore extending this approach to other conference types and criteria, as well as improving the accuracy of AI assessments through feedback loops. Additionally, as AI-generated submissions become more sophisticated, pre-review systems will need to evolve to detect potential biases or gaming.
6. Conclusion
We have presented an experience report on using generative AI for pre-reviewing conference submissions at BOSC 2026. Our tools, bosc-pre-review and Runabilly, effectively assisted volunteer reviewers by automating evidence gathering, while preserving human decision-making. This case study offers insights for other conferences facing similar challenges, highlighting the potential of AI to enhance—not replace—human expertise.
Comments: 18 pages, 4 figures
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL)
Cite as: arXiv:2607.27228 [cs.CL]
via ArXiv CL