I was confused when i was reading about weight pruning on CNN. Is it applied for all the layers including convolutional layers or only it is done for dense layers?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Here is a more complete articlerelated to your question.
The idea of pruning is simple and logical, however when looking at the bigger picture it is not as straight forward to implement. One of the main reason is that operations on the deeper layer depends on the previous layers and hence tampering with early layers might affect the next layers. Hence, if you were to prune one layer in the middle the following layers will have in some way pruned as well (I am not sure about the detail how to correctly implement this).
As discussed above it is quite a pain to implement correctly ( and hence not a recommended practice_. A better alternative for this is knowledge distillation which is to make a smaller scale network but we train it such that it will mimic the output of a more complicated teacher model. There are tons of good example where this works e.g. Distillbert. Here is a nice link related to this which also contains tons of reference.
$\begingroup$
$\endgroup$
The short answer is both.
Filter pruning
One approach is to remove filters that are "less contributing" to the overall learning every epoch or so. Defining "less contributing", that's the interesting bit about this research. For example it could be based on the running mean average of each filter, and remove filters of CNNs that have smaller mean. Some other research out there,
- Using L1 norm to prune filters paper
- This article list several other approaches (https://jacobgil.github.io/deeplearning/pruning-deep-learning)
Dense layer pruning
Obviously the parameter bottleneck of pretty much all CNNs happens at the first dense layer connected to the output of the last convolution layer. Therefore, doing proper pruning on this layer will be quite effective.
I couldn't find any good papers that introduces parameter compression for dense layers. But I'll update my solution if I come across any.
Knowledge distillation
Another really cool approach is known as knowledge distillation. Knowledge distillation is when you train a smaller network to mimic a larger network. One popular paper is found here.
