How does training a ConvNet with huge number of parameters on a smaller number of images work?

Question

I have two questions:

1. I am wondering why is that a very deep model such as VGG-16 which has approximately 138 million parameters (Source) can be used as a model to be trained on just 1.3 million images? Even though the authors of VGG-16 used dropout and regularisation to avoid overfitting (source), I just feel that the amount of parameters is still significantly more than the number of images.

2. Is it also true that when you use $N$ number of epochs, you "effectively" have $K*N$ training datasets, where $K$ is the number of training datasets you have? In the case above, $K$ = 1.3 million. Personally I don't think this is the case, but I am not sure.

I probably have not done research or read thoroughly, so I would like to apologize if the answers are already somewhere out there. Thank you.

Answer 1

First of all, you are ignoring the dimensionality of the problem. Images are very high-dimensional. Let's say an image has a resolution of $256 \times 256$, which means each image has $65,536$ pixels. ImageNet images are RGB, so each image has 3 channels, resulting in $196,608$ pixels per image. Now, the whole dataset ($1.3$m images) has more than $255$ billion pixels associated with its images.

Because filters work at pixel-level, the information it gains from one image is more than what a regular ML algorithm would gain from a single training example.

As you can see, the kernel (whose weights you want to update, and accounts to 9 parameters in this case) is much smaller than the input image. It would be a mistake to consider a single image (of around $200,000$ pixels as we say previously) as only capable of updating one parameter.

Secondly, the input of each layer changes from layer to layer, because of the sequential nature of the network. The second layer sees the output of the first,and so on... This means that the second layer's filters won't see the same image as the first and in fact, as training progresses, their input will also gradually change (because the first layer's filters will be getting more effective).

By the time it reaches let's say the $M^{th}$ layer, the input will have changed a lot from the original image, so it would be a mistake to consider that the original image is updating the totality of the parameters in the network.

Thirdly, data augmentation is also a thing. By making simple random transformations to each image (flips, shifts, scales, rotations, brightness/contast adjustments etc.) the network is tricked into thinking that this is a totally new image. This can exponentially increase the size of the dataset.

Finally, you should take a look at where the parameters in a network are. The VGG19 architecture does have more than $143$ million, but around $124$m of those are from the last 3 FC layers. In fact the first of the three layers has around $103$m parameters on its own! This is a very inefficient network design and that the research community has strayed far from recently. A more representative network you should look at is the ResNet architecture. For example a 50-layer ResNet (much deeper than the 19-layered VGG) consists of just around $25$m parameters, while achieving a similar -if not better- performance.