ScarfBench is a benchmark designed to evaluate AI agents on the task of migrating enterprise Java applications between frameworks. Developed by IBM Research, it addresses the growing need for automated tools that can handle large-scale, complex code transformations in production environments. As of May 2026, the dataset is hosted on Hugging Face under the repository ibm-research/ScarfBench and has garnered 492 downloads and 17 stars, reflecting early interest from the research and engineering community.
The benchmark focuses on real-world migration scenarios, such as moving from legacy frameworks like Java EE or Struts to modern alternatives like Spring Boot or Quarkus. It provides a structured evaluation framework to assess an AI agent's ability to understand code semantics, preserve business logic, and refactor dependencies without introducing errors. Key metrics include correctness, completeness, and migration efficiency.
With the ongoing shift toward cloud-native and microservices architectures in enterprise Java ecosystems, ScarfBench plays a critical role in advancing AI-powered code migration. It supports reproducible experiments and is expected to guide the development of next-generation developer tools that reduce manual effort and migration risks.
For more details, visit the ScarfBench dataset on Hugging Face.