NAM Pruning: Enhance Model Pruning via Network Activations Minimization

Abstract

Deep neural networks have become an essential tool in machine learning due to their excellent performance. However, their sheer amount of parameters can lead to their infeasible application in limited hardware contexts. In this work we propose a novel formulation to enforce sparsity in model activations, decreasing the number of firing neurons to facilitate pruning methods in achieving better results. Our approach aims at reducing the model’s activation density by minimizing their l0 pseudonorm. Although l0 is the most suitable objective function to approximate this value, it is a nonconvex and discontinuous function for which optimization is NP-hard. To achieve the desired behavior we employ a differentiable and unbiased estimate of the actual l0 pseudonorm. Thanks to this estimate we are able to formulate a novel model training objective function that aims to minimize the empirical risk loss by adding sparsity in the model activations. To prove the suitability of our training method we define an iterative pruning schema designed to enforce sparsity in model activations during the pruning steps. Finally, to assert our formulation, we propose our iterative pruning algorithm which gives the ability to define a trade-off between final models accuracy and size. We compare it with two well-known trimming approaches, showing how it can lead to better pruning performance. We empirically assess the effectiveness of our method on two distinct DNNs trained on CIFAR10 and GTSRB.