Optimal Pruning for Neural Architectures Using Fisher Information Distances

arXiv:2609.16129 [cs.AI] | Submitted 14 Sep 2026

Authors: David S. Berman, Yen-Yu Fu, Edward Hirst, Thelma Chiwete Obirai

Subjects: Artificial Intelligence (cs.AI); Information Theory (cs.IT); Differential Geometry (math.DG)

Report Number: QMUL-PH-26-32

Comments: 21 pages, 4 figures, 4 tables

Abstract

A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in model space as determined by the Fisher information metric. This distance determines the true change in the model—and in its performance—under pruning.

By analyzing progressively more faithful approximations of this geodesic distance, a natural hierarchy of optimality for pruning methods is established. This hierarchy begins with traditional magnitude pruning and develops into new, more sophisticated and effective pruning schemes.

The method is demonstrated on both fully-connected networks and vision transformers, using MNIST and CIFAR-10, across the complete 0–100% pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation yields intermediate pruning schemes that are computationally efficient while maintaining near-optimal performance.

This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically motivated justification for pruning schemes.

Key Contributions

• Geometric framework for pruning: Reframes parameter pruning as a displacement to a vanishing hypersurface, with the minimal distance measured via the geodesic distance under the Fisher information metric.
• Hierarchy of pruning methods: Shows that traditional magnitude pruning corresponds to the crudest approximation of the geodesic distance, while more faithful approximations yield progressively better schemes.
• Empirical validation: Demonstrates consistent improvements over magnitude pruning and local Fisher information pruning across fully-connected networks and vision transformers on MNIST and CIFAR-10, over the full pruning range and across five random seeds.
• Efficiency trade-offs: Identifies intermediate approximations that remain computationally efficient while retaining near-optimal accuracy.

Citation

Berman, D. S., Fu, Y., Hirst, E., & Obirai, T. C. (2026).
Optimal Pruning for Neural Architectures using Fisher Information Distances.
arXiv:2609.16129 [cs.AI]. https://doi.org/10.48550/arXiv.2609.16129

via ArXiv AI