Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am studying the CNN architecture of the AlexNet, and I have seen that it has convolutional layers without pooling in between:
but I don' understand why this is done. Wouldn't be better to have something like CONV - POOLING - CONV - POOLING , and so on, instead of CONV - POOLING - CONV - CONV -CONV - POOLING?
Why is this done? Thanks in advance.
The idea behind consecutive convolutional layers with no pooling is actually not to skip any pooling but to replace a single layer with a bigger receptive field. So think of it this way:
Intuitively I like to think of it this way: For two 3x3 layers the actual receptive field gets extended by 1 in all directions as the information is "pulled in" from outside of the 3x3 field by applying two layers consecutively.
In the first layer the 3x3 receptive field is applied (here using A and B as examples):
Now, that means when the second 3x3 layer is being applied cell B contains information from beyond the dark grey area. And this information in B will be considered for A when you apply the second layer. So basically the receptive field of A has grown as it includes information from beyond its actual 3x3 field:
And since this happens in all directions as shown in the second picture you now end of with a sort of 5x5 receptive field. Finally, if you apply this a third time (like AlexNet does with its 3 consecutive 3x3 layers) then the receptive field will be extended a second time by 1 in all directions, i.e. you get a 7x7 receptive field.
Question is why would you actually do this and not just use a single layer with a larger receptive field?
The paper Very Deep Convolutional Networks for Large-Scale Image Recognition gives two reasons:
First, we incorporate three non-linear rectification layers instead of a single one, which makes the decision function more discriminative.
[...]
Second, we decrease the number of parameters: assuming that both the input and the output of a three-layer 3 × 3 convolution stack has C channels, the stack is parametrised by $3(3^2C^2)=27C^2$ weights; at the same time, a single 7 × 7 conv. layer would require $7^2 C^2=49C^2$ parameters, i.e. 81% more. This can be seen as imposing a regularisation on the 7 × 7 conv. filters, forcing them to have a decomposition through the 3 × 3 filters (with non-linearity injected in between).
