Abstract
In 2011, IBM's Watson was akin to a sealed capsule of its era's queryable knowledge. Its DeepQA system triumphed over the strongest human Jeopardy! champions, but the knowledge that enabled this feat resided in a curated billion-document corpus running on a cluster of POWER7 servers—frozen at build time and impossible to move or copy. We demonstrate that the same kind of artifact—a snapshot of what a culture can answer—is now portable and essentially free. We evaluate a single 9 GB open-weight model (Qwen2.5-14B, 4-bit) against the complete open Jeopardy! clue dataset, comprising 529,939 clues across all 41 broadcast seasons from 1984 to 2025. To our knowledge, this is the first time a model has been run over the full corpus. The 41 years mark only how long the questions were collected; what they test is far older and broader: the accumulated body of human general knowledge that a culture deems worth knowing—from ancient history and dead languages to science, literature, and geography—with a verified answer for every item. The model answers 67.0% of all clues under a strict forced-response protocol with exact and fuzzy matching, and it exceeds 85% on factoid categories. We treat training-data exposure as something both systems share rather than as a flaw unique to language models. Watson's case is, in fact, the more extreme one: its corpus was assembled to contain Jeopardy answers, it was tuned on past clues, and it could not answer anything outside that curated distribution. The decisive test is whether a model can handle clues that did not exist when it was built. On clues aired after its training cutoff, the local model holds 65% and Claude Opus 4.8 holds 95%, while Watson, by construction, scores zero. The capability survives the move from a server room to a file you could seal in a time capsule, and—unlike Watson—it is not frozen to its own moment.
1. Introduction
Since IBM's Watson first showcased the power of question answering on Jeopardy! in 2011, the landscape of artificial intelligence has shifted dramatically. What once required an array of dedicated servers and a massive, curated corpus can now be achieved with a single local model that fits on a standard desktop. This paper revisits the notion of a "time capsule of testable human knowledge," demonstrating that modern open-weight models can encapsulate not just a snapshot of a culture's knowledge but also a portable and freely accessible one.
We focus on Qwen2.5-14B, a 14-billion-parameter model quantized to 4-bit precision, which runs locally without specialized hardware. By evaluating it against the entire Jeopardy! archive—spanning 41 broadcast seasons from 1984 to 2025—we assess the breadth and depth of its general knowledge. This work is, to our knowledge, the first to run a single model over the entire corpus, providing a comprehensive benchmark for open-domain question answering.
2. Background and Related Work
2.1. Watson and the DeepQA System
IBM's Watson represented a landmark achievement in natural language processing and information retrieval. Its DeepQA pipeline combined multiple algorithms to generate, analyze, and rank hypotheses from a curated corpus of over 10 million documents. The system was optimized for Jeopardy!'s specific format, requiring rapid, confidence-weighted responses. However, Watson's knowledge was static; it could not learn from new information after its deployment, and its corpus was meticulously constructed to include potential answers.
2.2. The Evolution of Open-Weight Models
The years following Watson witnessed the rise of large language models (LLMs), culminating in open-weight releases like the Qwen series. These models, trained on diverse text corpora, exhibit remarkable generalization abilities. Quantization techniques allow models like Qwen2.5-14B to run efficiently on consumer hardware, bridging the gap between high performance and accessibility. Unlike Watson, these models can be updated and deployed across a myriad of tasks, making them versatile tools for knowledge retrieval.
2.3. Jeopardy! as a Benchmark
Jeopardy! remains a unique benchmark for general knowledge, testing not only factual recall but also the ability to parse wordplay, synonyms, and domain-specific terminology. The complete archive, released openly, offers a vast and varied dataset for evaluating models. Previous studies have used subsets or filtered clues, but none have attempted a full-corpus evaluation until now.
3. Methodology
We employed the Qwen2.5-14B model, quantized to 4-bit precision, running locally via the llama.cpp library. The entire Jeopardy! archive was fed as input, with clues formatted as: "Question: {clue} Answer: {the}". The model was constrained to a forced-response protocol, generating a single answer per clue. We evaluated responses using exact string matching and fuzzy matching (employing a similarity threshold of 0.8). Additionally, we categorized clues into factoid (e.g., geography, history) and non-factoid (e.g., wordplay, avant-garde questions) types to analyze performance variations.
For comparison, we benchmarked the same dataset against Claude Opus 4.8, a state-of-the-art closed-source model accessed via API, and against Watson's published results (examined as a baseline). We also split the dataset based on broadcast date to assess post-training-cutoff performance (clues aired after Qwen's training data cutoff of mid-2024).
4. Results
4.1. Overall Performance
Across all 529,939 clues, Qwen2.5-14B achieved an accuracy of 67.0% under strict matching (exact + fuzzy). Factoid categories, such as science, history, and geography, yielded accuracies exceeding 85%, while non-factoid categories, particularly those involving wordplay, dropped to around 50%. This performance is notable given the model's size and resource footprint.
4.2. Temporal Analysis
For clues aired before the model's training cutoff (mid-2024), accuracy was 67.5%. For clues aired after the cutoff, accuracy held at 65.0%, demonstrating robustness to temporal generalization. In contrast, Claude Opus 4.8 achieved 95% on post-cutoff clues, while Watson scored zero (as it had no access to unseen information).
4.3. Comparison with Baseline
Compared to historical benchmarks, Qwen2.5-14B outperforms earlier LLMs and rivals some larger closed models on this task. Its local operation capability and free availability make it an attractive alternative for knowledge-intensive applications.
5. Discussion
5.1. Training Data Exposure: Shared Reality, Not Flaw
Both Watson and modern LLMs are trained on historical data, and their performance on Jeopardy! reflects this. Critics often cite training data exposure as a limitation of LLMs; however, Watson's corpus was itself constructed to contain Jeopardy answers, making its advantage more engineered. We argue that exposure to the dataset is not a disqualifying factor but a shared characteristic that both systems exhibit.
5.2. The Decisive Test: Post-Cutoff Generalization
The ability to answer clues unseen during training separates adaptive models from static ones. While Qwen2.5-14B declines slightly on post-cutoff clues, its 65% accuracy indicates robust generalization. Claude Opus's 95% suggests that larger, continually updated models can maintain high performance, but even a modest local model retains significant capability. This is a marked improvement over Watson, which by design could not adapt.
5.3. Practical Implications
The portability and low cost of models like Qwen2.5-14B open new avenues for knowledge preservation. A 9 GB file can encode a substantial portion of human general knowledge, accessible offline and indefinitely. This "time capsule" property has profound implications for archival, education, and AI accessibility.
6. Conclusion
This study demonstrates that a single, free, local open-weight model can encapsulate decades of testable human knowledge, achieving 67% accuracy on the complete Jeopardy! archive. The model's performance, especially on post-training data, underscores the evolution of AI from server-bound behemoths to portable knowledge repositories. While Watson was frozen in its moment, contemporary models offer a dynamic and accessible alternative, preserving not just facts but the potential for continued learning. Future work should explore larger open models, fine-tuning for specific domains, and the integration of such models into everyday knowledge systems.
via ArXiv AI