Abstract
Text classifiers trained on the ISOT/Kaggle "Fake and Real News" corpus routinely report accuracy and F1 above 0.98 โ a level of performance that sits uneasily beside the inherent difficulty of assessing veracity. Using a transparent TF-IDF and linear-classifier pipeline as a measurement instrument, this study audits the corpus along three leakage channels and two distribution-shift protocols, releasing all code and derived numbers.
Key findings
- Partly degenerate benchmark. A classifier given only the
subjectmetadata field โ with the article text discarded โ attains F1 = 1.000, because the two classes have disjoint subject values. - Residual signal is diffuse style, not telltale tokens. Removing all three leakage channels โ metadata, a newswire source tag present in 99.2% of real articles, and 6,251 duplicate documents contaminating 19.4% of a naive test split โ lowers F1 by only 1.21 points (0.9935 โ 0.9814). The remaining signal is diffuse editorial style rather than a few giveaway tokens: deleting the 1,000 highest-weight unigrams still leaves F1 = 0.926.
- The style signal does not transfer. Under a topic-disjoint protocol, average precision falls from 0.9995 to 0.9475 and deployed F1 from 0.9905 to 0.8067, with a prior-matched analysis confirming a genuine 5.2-point loss of discrimination. Temporal transfer, by contrast, is nearly lossless.
- Added capacity exploits the shortcut. A fine-tuned DistilBERT is stronger in-distribution (F1 = 0.9993) but degrades far more under topic shift, losing 12.9 average-precision points versus the linear model's 5.2.
- No cross-benchmark transfer. Transferred to the independent LIAR benchmark, all three models fall to near-chance ranking (ROC-AUC 0.54โ0.57), none beating a majority-class baseline.
Conclusion
Within-corpus scores on this dataset quantify source and topic separability rather than veracity. Added model capacity exploits the shortcut rather than avoiding it. The authors recommend metadata-only, small-sample, and topic-disjoint baselines as inexpensive diagnostics for future work.
Paper Metadata
- arXiv ID: arXiv:2609.25006 [cs.CL]
- Submitted: 26 Jul 2026
- Author: Yuvraj Verma
- Primary subject: Computation and Language (cs.CL)
- Cross-list: Machine Learning (cs.LG)
- ACM classes: I.2.7; I.5.4
- Comments: 17 pages, 11 figures, 11 tables. Code, experiment scripts, and machine-readable results: github.com/vermayuvraj/fake-news-detection
- Version: v1 (this version)
Why This Matters in 2026
As misinformation detection systems are increasingly deployed in production โ from platform moderation pipelines to newsroom verification tools โ benchmark scores remain the default proxy for trustworthiness. This audit lands at a moment when the field is reckoning with a broader reproducibility crisis in NLP evaluation. Recent surveys of dataset contamination (e.g., work on n-gram overlap between pretraining corpora and downstream benchmarks) and of shortcut learning have repeatedly shown that headline numbers can be inflated by artifacts rather than genuine capability. The ISOT corpus, one of the most widely cited fake-news datasets, has anchored thousands of papers reporting >0.98 F1; this study shows those numbers largely reflect source and topic separability, not veracity detection.
The implications extend well beyond one dataset:
- Evaluation hygiene. Metadata-only and topic-disjoint baselines are cheap, fast, and revealing. They should be standard practice, not an afterthought.
- Model scaling caveat. Larger models (here, DistilBERT) amplify shortcut exploitation rather than mitigating it โ a cautionary note as the field leans further into pretrained and LLM-based classifiers in 2026.
- Cross-dataset generalization. Near-chance ROC-AUC on LIAR underscores that in-distribution gains do not translate into real-world robustness.
All code, experiment scripts, and machine-readable results are publicly released, making this audit directly reusable as a diagnostic template for other corpora.
via ArXiv CL+LG