Zugwise: Domain-Aware Game Reconstruction for Digitizing Handwritten Chess Scoresheets

Author: Gerhard Trippen (University of Toronto)

Subjects: Artificial Intelligence (cs.AI); Image and Video Processing (eess.IV)

arXiv: 2611.14544v1 [cs.AI] — Submitted 26 October 2026

Comments: 54 pages, 3 figures. Code and models available on GitHub.

Abstract

Digitizing handwritten chess scoresheets has traditionally been framed as a handwriting-recognition problem: read the move in each box and verify that it is legal. This framing overlooks the errors that matter most. A misread producing an illegal move is conspicuous. A misread producing a legal-but-wrong move is silent, because a legality-only pipeline accepts it and then reads every subsequent move against an incorrect position.

We therefore treat the task as game reconstruction and present Zugwise, a system that repurposes computer-chess machinery—quiescence search, static exchange evaluation, and pin detection—to ask not "what should be played?" but "could a human plausibly have written this?" An absurdity detector makes silent errors loud at or near where they occur, while a backward search traces late symptoms to their cause. Candidate ranking deliberately excludes chess-quality signals. A Needleman-Wunsch merge incorporates the second player's sheet as corroborating evidence, and review stops route every disagreement between the two sheets' legal readings to a human.

In oracle replay of two tournament sections, a release incorporating these mechanisms cuts silent errors from 16 to 1 and from 50 to 13. Two learned signals—CTC re-scoring against the recognizer's raw output and a Maia-style human-move prior—raise the correct fix's top-rank rate from 58.8% to 78.0% of replacement decisions where it is a candidate, and hold without refitting on a later section. On the first section, the human-verified output matches an independent expert transcription move for move on 34 of 45 games; adjudication against the sheets found errors in both records, on different plies. We do not establish that either workflow is faster or more accurate.

The system runs entirely in the browser and is free, private, and offline-capable. Alongside it, we offer a three-way adjudication protocol for a domain in which no single record is ground truth.

Key Contributions

• Game reconstruction framing replaces legality-only handwriting recognition, surfacing silent legal-but-wrong errors.
• Domain-aware verification adapts quiescence search, static exchange evaluation, and pin detection into an absurdity detector with backward error tracing.
• Dual-sheet fusion via a Needleman-Wunsch merge uses the opponent's scoresheet as independent evidence.
• Learned re-ranking combines CTC re-scoring and a Maia-style human-move prior, improving correct-fix top-rank rate from 58.8% to 78.0%.
• Deployment: a free, private, offline-capable browser-based tool.
• Adjudication protocol for domains without a single ground-truth record.

2026 Context

As of 2026, handwriting recognition for structured documents has matured through large multimodal models, yet domain-specific verification remains essential where downstream reasoning compounds errors. Zugwise exemplifies a broader trend: pairing perception models with symbolic, domain-grounded consistency checks rather than relying on end-to-end recognition alone. Its browser-native, offline design also aligns with growing demand for privacy-preserving, on-device AI workflows.

Resources

• Code and models: GitHub — GerhardTrippen/Zugwise
• DOI: https://doi.org/10.48550/arXiv.2611.14544

via ArXiv AI