Less parameters - in general within ResNets

Question

My question is about the parameters of the ResNet.

Why does the network tend to have fewer parameters than the VGG? This would be the case if I got the paper and the summary from Yannic Kilcher correct.

As far as I understood it, you concatenate the input x with the output x_prime of another layer in a residual block. This enables you to train more stable networks even if you go deeper.

Why does this lead to fewer parameters than the VGG? I would suggest that this is the case because you perform the more costly operations (more filters) on layers already reduced in their dimensions due to the stride? Is this correct?

Answer 1

In the aforementioned image, we can see that even if Resnet-34 has more Convolutional layers, it still has 7-8 times fewer parameters and FLOPs than VGG-19.

Clearly, Convolutional layers are not at fault. But fully connected layers are!! In VGG-19 there are 3 big fully connected layers after the backbone. On the other hand, Resnet has a global average pooling layer which dramatically reduces the size of output (H and W dimensions) from the backbone. And following it there is only one fully connected layer. The global average pooling trick saves a lot of parameters and hence the absence of this layer in VGG results in a very big output from the backbone. And as a result, VGG needs wider fully connected layers, and adding more such layers again adds up lots of parameters.

Answer 2

There are 2 different levels of complexity in a network :

1. Number of parameters
2. Number of operations (FLOPs)

It is especially important to make a distinction when using CNN since a convolution kernel is applied on many different pixels, so a same weights will be used in different computations. The ratio $operations/parameters$ is approximately $1$ in a fully connected network, but in a CNN it is way more important.

About the number of parameters

This article explains very well the number of parameters of each CNN architecture, you should give it a look.

If you take a look at the tables of parameters of ResNet and VGG, you will notice that most of VGG parameters are on the last fully connected layers (about 120 millions of the 140 millions parameters of the architecture). This is due to the huge size of the output layer of the convolutional part.

The output size is 512 7*7 features maps, so it is the equivalent of a $512*7*7 = 25088$ size layer in a fully connected network. This is why the connection to the next layer (which has 4096 neurons) is very expensive and requires $25088*4096 = 100M$ parameters.

This is quite a particularity of VGG, this architecture has 70% of its parameters used for one layer.

Now if you compare with ResNet, ResNet use an avg pool on each feature maps at the end, so the number of outputs of the convolutional part is 512 values, which leads for fully connected part of the network to have $1000*512 = 512 000$ parameters (if we forget about the bias of each neuron).

This is why (at least in my brain) VGG has so many parameters while ResNet keep it fairly low.

About the number of operations (FLOPs)

As I mentionned previously, a layer with many parameters may not be the hardest to compute for the network, it depends if the parameters is only used once (fully connected layers) or more (convolutional layers), so the 100M parameters layer of VGG may not be the cause of its important number of operations.

I'm way less confident in my explanation here, but I think ResNet efficiency is due to the fact that Resnet quickly diminishes the size of feature maps (first layer divide it by 2 and second layer as well), so since we work on less pixels, we have less computations to do.

VGG applies 3x3 convolution kernels on each of the 224*224 pixel of the image on all 3 of its first layers, which is what is not efficient and requires a lot of computational power.

That is how I would explain ResNet having less computational complexity than VGG

Sorry for writing something so long, hope it helps.