Neural Network Pruning Optimized via Fisher Information Distances

VERA Brief

AI-generated. Grounded in the article and its cited sources.

A new parameter pruning method uses differential-geometric distances in model space, calculated via the Fisher information metric, to find optimal pruning strategies. This approach analyzes geodesic distances to quantify model changes and performance post-pruning, outperforming existing methods.

Key facts

• A novel parameter pruning methodology is introduced, based on differential-geometric distances in model space.
• The method models parameter pruning as a displacement to a hypersurface where the parameter value is zero.
• Geodesic distance, defined by the Fisher information metric, is used to calculate the minimal distance to this hypersurface.
• The approach has been demonstrated on fully-connected networks and vision transformers using MNIST and CIFAR-10 datasets.
• The method reportedly outperforms parameter magnitude pruning and local Fisher information pruning in terms of accuracy and Matthews correlation coefficient.