Large Language Model (LLM) agents have demonstrated significant potential for automating data-science workflows, but their end-to-end performance is critically dependent on the agent harnessβthe framework that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback. In many existing data-science agents, this harness remains implicit, which hampers reproducibility, comparability, and attribution of results across heterogeneous tasks. In response, we introduce DS-Lighting, a unified harness toolkit that makes harness design explicit for data-science automation.
DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation. It represents diverse agents as executable operator programs that support both predefined pipelines and adaptive search strategies. Furthermore, we integrate multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparisons under a shared task interface, a sandboxed runtime, and a standardized metric protocol.
Through extensive experiments across various agents, harnesses, models, and ablations, we demonstrate that explicit harness design significantly improves reproducibility, comparability, and reliability, while reducing avoidable system-level failures in end-to-end data-science workflows. As the field moves toward more autonomous data-science agents in 2026, DS-Lighting provides a crucial foundation for transparent and robust agent evaluation. Our code is publicly available at: https://github.com/usail-hkust/dslighting.
via ArXiv AI