0
$\begingroup$

I was confused when i was reading about weight pruning on CNN. Is it applied for all the layers including convolutional layers or only it is done for dense layers?

$\endgroup$
1
$\begingroup$

Here is a more complete articlerelated to your question.

The idea of pruning is simple and logical, however when looking at the bigger picture it is not as straight forward to implement. One of the main reason is that operations on the deeper layer depends on the previous layers and hence tampering with early layers might affect the next layers. Hence, if you were to prune one layer in the middle the following layers will have in some way pruned as well (I am not sure about the detail how to correctly implement this).

As discussed above it is quite a pain to implement correctly ( and hence not a recommended practice_. A better alternative for this is knowledge distillation which is to make a smaller scale network but we train it such that it will mimic the output of a more complicated teacher model. There are tons of good example where this works e.g. Distillbert. Here is a nice link related to this which also contains tons of reference.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The short answer is both.

Filter pruning

One approach is to remove filters that are "less contributing" to the overall learning every epoch or so. Defining "less contributing", that's the interesting bit about this research. For example it could be based on the running mean average of each filter, and remove filters of CNNs that have smaller mean. Some other research out there,

Dense layer pruning

Obviously the parameter bottleneck of pretty much all CNNs happens at the first dense layer connected to the output of the last convolution layer. Therefore, doing proper pruning on this layer will be quite effective.

I couldn't find any good papers that introduces parameter compression for dense layers. But I'll update my solution if I come across any.

Knowledge distillation

Another really cool approach is known as knowledge distillation. Knowledge distillation is when you train a smaller network to mimic a larger network. One popular paper is found here.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.