I am currently reading about Convolutional Neural Networks in Deep Learning Book. I am stuck on section 9.4 titled "Convolution and Pooling as Infinitely String Priors".
Could someone please intuitively explain to me what prior probability distribution is and what is its association to Convolution and Pooling operations in context of CNNs. Thank you!
A prior distribution expresses your assumptions about the model without observing any data. E.g. when doing linear regression, you a priori assume that the slope is close to zero. Now you start measureing data points and it turns out that the slope should probably be close to one, so you compromise and pick a value somewhat inbetween. If your belief in your prior assumption of zero slope is weak, it does not take much data to convince you to pick a slope closer to one. If your belief is strong, you will pick a slope close to zero anyway and demand to see many data points until you move it slowly towards one.
In case of regression the strength of your belief would be parameterized by the weight of the regularization term in the optimization objective. The connection between regularization and priors is the same for neural networks (see here for a little more detail).
So you can think of the strength of the prior as a measure for how much evidence you would need to see until you deviate from your prior assumptions about the model.
When you train a CNN, you assume a priori that this network structure is the best for your problem, i.e. that the model should internally compute convolutions. Since the CNN structure is build into the model, no amount of data will convince you to abandon this structure in favour of e.g. a fully-connected NN. Hence the prior is infintely strong.
The argument for pooling etc. would be similar. I would even claim that most if not all non-trainable parameters can be seen as an infinitely strong prior.
