Counting the Bandages: Measuring the Time AI Actually Saves

arXiv:2610.35942 [cs.AI] — Submitted 23 October 2026

Author: Solomon Shalom Lijo

Subjects: Artificial Intelligence (cs.AI)

Comments: 18 pages, 2 figures. Replaces the author's earlier TSB proposal (SSRN 6682338). Code and data are available on GitHub and archived on Zenodo.

Abstract

This paper examines what it actually means for AI assistance to save time. The author contrasts a stopwatch view—in which time saved is simply whatever the clock records—with a ledger view, in which time saved is a counterfactual that must be read alongside the errors the work lets through. The paper argues for the latter.

The argument proceeds in four parts:

1. A model-free measurement framework. The author defines a pair of quantities—time saved and excess escaped defects—and derives a single decision number from them: the break-even defect cost.


1. A baseline model of checking. Until a model is good enough that checking becomes unnecessary, a calibrated worker's escaped-defect rate does not depend on how good the model is. Checking time grows with the logarithm of the model's error rate, and over-trust buys minutes on the clock at a larger cost in total minutes.


1. What randomized studies can and cannot identify. The author replicates METR's developer study from its public data. The same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them.


1. A restatement of the published record, with stated bounds replacing imported point assumptions. Within GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6×; consultants outside the frontier were faster—and more often wrong.



Finally, the author tests two of the model's predictions in a small, pre-specified experiment on AI reviewers checking AI drafts of the same kind of brief, built from SEC filings made after the models' training data were collected. When told the drafter's true error rate, the reviewer checked more as the rate rose—but all-or-nothing, switching near where its prompt's costs break even. When told the drafter erred ten times less often than it actually did, the reviewer checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none: faster and worse, entirely through reduced effort.




---




Key Contributions



• Stopwatch vs. ledger framing for evaluating AI time savings, with a formal case for the ledger view.
• Model-free metrics: time saved, excess escaped defects, and break-even defect cost.
• A baseline checking model linking error rates, checking time, and over-trust costs.
• Replication of METR's developer study, quantifying estimand sensitivity in the reported slowdown.
• Reanalysis of GDPval and GPT-5 workflows, showing how review time caps achievable speedups.
• A pre-specified experiment demonstrating that miscalibrated reviewer beliefs about drafter error rates lead to reduced checking effort—and more escaped errors.



---




Resources



• Code and data: github.com/solomonshalom/tsb-bench
• Zenodo archive: doi.org/10.5281/zenodo.23557773
• arXiv abstract: arXiv:2610.35942

via ArXiv AI