Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
Authors: Ruoxi Zhao and Maziar Raissi
Abstract
Evaluating large language model (LLM) coding agents in algorithmic trading presents unique challenges. Static benchmarks risk data contamination, while numerical backtest outputs require ground truth derived from actual code execution. To address these issues, we introduce Backtrader-Bench, a framework employing two complementary pipelines. The deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations spanning five trading strategies, 33 templates, and three difficulty tiers, with an independent checker re-deriving every answer. The generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them into MCQs, and discards any that a no-tool solver can answer without code execution. We evaluate 11 models without tools (10 runs each) and four tool-augmented configurations on a curated set of 30 questions. Tool-augmented agents achieve 90.0% accuracy in a single pass (GPT-5.5 and Opus 4.7), outperforming the best no-tools baseline (73.0%, averaged over 10 runs) by 17 percentage points. On 38 separately mined questions, no-tools accuracy drops further, with half of the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to support reinforcement learning training corpora, aiming ultimately to build a specialized agent for quantitative trading workflows.
Introduction
The rapid advancement of large language models (LLMs) has opened new frontiers in algorithmic trading, where coding agents can analyze market data, implement strategies, and execute backtests. However, evaluating these agents remains problematic. Conventional static benchmarks often suffer from data contamination, where models have memorized test samples during training. Additionally, trading algorithms' numerical outputs—such as backtest performance metrics—require ground truth generated through actual code execution, complicating validation.
To overcome these limitations, we present Backtrader-Bench, a novel benchmarking framework that leverages self-generated multiple-choice questions. This approach generates questions dynamically, ensuring freshness and reducing contamination risk. Each question includes an independent verification step, making the evaluation both deterministic and grounded in executable code.
Our framework offers two complementary pipelines:
- Deterministic MCQ Pipeline: Generates questions from backtest configurations across five trading strategies, using 33 templates and three difficulty tiers. An independent checker re-derives every answer, ensuring correctness.
- Generator-Solver Filtering Pipeline: Autonomously mines harder questions. A generator creates questions verified by executable code, converts them into MCQs, and discards any that a no-tool solver can answer without code execution.
We evaluate 11 models without tools (10 runs each) and four tool-augmented configurations. Results show that tool-augmented agents achieve substantially higher accuracy, demonstrating the value of code execution in complex trading scenarios.
Methodology
Deterministic MCQ Pipeline
The deterministic pipeline formalizes backtest configurations into structured templates that generate MCQs. Each question is tied to a specific trading strategy and difficulty tier. An independent checker validates answers by re-running the corresponding code, ensuring that every answer is correct and reproducible.
Generator-Solver Filtering Pipeline
This pipeline autonomously identifies challenging questions. A generator produces questions verified by executable code, then converts them into MCQs. The solver component filters out questions that can be answered without code execution, retaining only those requiring deep reasoning or tool use.
Experimental Setup
We curated 30 questions for the primary evaluation and 38 separately mined questions for additional testing. Eleven models were evaluated without tools (10 runs each) to capture performance variability. Additionally, four tool-augmented configurations (e.g., GPT-5.5 and Opus 4.7 with code execution) were tested.
Results
Tool-augmented agents achieved 90.0% accuracy in a single pass, a 17-point improvement over the best no-tools baseline (73.0% averaged over 10 runs). On the harder 38-question set, no-tools accuracy dropped significantly, with half of the models falling to around random chance (25%). These results underscore the importance of code execution for tasks requiring exact calculations or logic.
Discussion
Backtrader-Bench offers a scalable evaluation framework that mitigates data contamination and enhances reliability. Its MCQ infrastructure can also generate training corpora for reinforcement learning, potentially leading to specialized agents for quantitative trading. Future work will expand strategies and templates, and explore cross-domain generalization.
Conclusion
We introduced Backtrader-Bench, a robust benchmark for LLM agents in algorithmic trading. By combining deterministic pipeline generation with autonomous question mining, we provide a reliable and contamination-resistant evaluation method. Our findings highlight the critical role of tool augmentation in achieving state-of-the-art performance in complex trading tasks.
Availability
Code and data are available at this https URL. The work has been accepted to the FinLLM Workshop at IJCAI 2026.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2608.11232 [cs.CL]
DOI: https://doi.org/10.48550/arXiv.2608.11232
via ArXiv CL+LG