Summary
Evaluating wildfire risk systems using standard machine learning metrics—such as F1-score or Intersection over Union (IoU)—is fundamentally flawed. These metrics assess event prediction accuracy, not the operational coherence of a continuous risk signal. This paper proposes a novel monotonic evaluation framework that measures whether increases in a predicted risk score consistently correspond to increases in observed operational load, including number of fires, intervention time, and deployed resources.
Methods and Approaches
The study compares three structurally different approaches applied to the French Alpes-Maritimes department:
- DFE index: An expert-based system.
- GRU-based predictive models: Deep learning models using Gated Recurrent Units.
- FARS: A hybrid multi-agent system combining predictive AI with LLM-based reasoning.
Key Findings
Experimental results reveal that the DFE index, despite having poor classification metrics, exhibits the most balanced monotonic behavior across the full risk scale. GRU models achieve strong local monotonicity but fail to produce well-distributed risk levels. FARS inherits and reveals the structural limitations of upstream signals rather than correcting them.
Central Paradigm Shift
The central finding is a paradigm shift: a good risk model does not predict fires accurately. Instead, it produces an ordinal scale that meaningfully explains operational dynamics, as demonstrated in this paper.
Code Availability
The code for the monotonic framework is publicly available on GitHub.
Publication Details
This work has been accepted for presentation at the 2026 IEEE 50th Annual Computers, Software, and Applications Conference (COMPSAC). The preprint is available on arXiv (arXiv:2607.21597) and was submitted on April 23, 2026.
via ArXiv AI