DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

Authors: Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang

Submitted: 19 May 2026

Abstract

The quality of training data fundamentally determines the capabilities of large language models (LLMs). Yet, as of 2026, no unified benchmark exists to comprehensively measure how well LLMs, AI agents, and data-centric workflows prepare training data from end to end. We conceptualize LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training utility of candidate datasets before downstream fine-tuning. Throughout, “quality” refers to downstream training effectiveness rather than surface-level textual properties.

We introduce DataPrep-Bench—the first unified benchmark that jointly evaluates both capabilities under a shared, downstream-grounded protocol across six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs in conjunction with the Dolly-15k dataset. As part of this track, we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 absolute points on Llama-3.1-8B in the Finance domain, and proves competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains.

For data quality evaluation, scoring functions are assessed by Pearson correlation with downstream performance on a shared candidate pool. We release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses the Maximum Mean Discrepancy (MMD) between a candidate dataset and a domain proxy. DAS achieves the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical domains, outperforming existing quality-, diversity-, and heuristic-based evaluators.

DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both data construction and quality evaluation as co-equal targets in LLM-driven data preparation. The benchmark is designed to scale with future LLM capabilities and data preparation workflows in 2026 and beyond.

via ArXiv LG