The idea behind consecutive convolutional layers with no pooling is actually not to skip any pooling but to replace a single layer with a bigger receptive field. So think of it this way:
- Two consecutive 3x3 layers actually lead to a receptive field of 5x5.
- Three consecutive 3x3 layers (like in AlexNet) lead to an actual receptive field of 7x7.
Intuitively I like to think of it this way: For two 3x3 layers the actual receptive field gets extended by 1 in all directions as the information is "pulled in" from outside of the 3x3 field by applying two layers consecutively.
In the first layer the 3x3 receptive field is applied (here using A and B as examples):
Now, that means when the second 3x3 layer is being applied cell B contains information from beyond the dark grey area. And this information in B will be considered for A when you apply the second layer. So basically the receptive field of A has grown as it includes information from beyond its actual 3x3 field:
And since this happens in all directions as shown in the second picture you now end of with a sort of 5x5 receptive field. Finally, if you apply this a third time (like AlexNet does with its 3 consecutive 3x3 layers) then the receptive field will be extended a second time by 1 in all directions, i.e. you get a 7x7 receptive field.
Question is why would you actually do this and not just use a single layer with a larger receptive field?
The paper Very Deep Convolutional Networks for Large-Scale Image Recognition
gives two reasons:
First, we incorporate three non-linear
rectification layers instead of a single one, which makes the decision function more discriminative.
Second, we decrease the number of parameters: assuming that both the input and the output of a three-layer 3 × 3 convolution stack has C channels, the stack is parametrised by $3(3^2C^2)=27C^2$ weights; at the same time, a single 7 × 7 conv. layer would require $7^2 C^2=49C^2$ parameters, i.e. 81% more. This can be seen as imposing a regularisation on the 7 × 7 conv. filters, forcing them to have a decomposition through the 3 × 3 filters (with non-linearity injected in between).