Transfer learning on new image size

Question

Transfer learning: Take a trained neural network and use it for a new classification task.

When we want to use transfer learning with a convolutional neural network, we don't have to use the same image size as input than the image size used for training. But if we change the input size we will have to re-train fully connected layers. See this post on Stackoverflow.

I don't undertand why changing input shape will not affect convolutional layer weight and why it will affect fully connected layer weights.

Tell me if my question is not understanble.

Answer 1

To understand this you need some basic mathematics of how matrix operations work. So let us start, consider you are using resnet-34 architecture which is trained on imagenet with 1000 classes so when you use tranfer learning you load the model architecture and its weights a model like resnet34 has two things backbone that is the convolution part and FCL's the neck of the network now when you train a model using resnet 34 it will only train the neck part now let us understand, convolutions work on whole images they are filters of some nxn dimension which work on a image of suppose dxk dimension now convolution filter sizes remain the same what changes is the feature map size now if you see usisg a nxn filter on dxk image will give you a feature map of (d-n+1)x(k-n+1) dimension for that particular filter now for the other convolution units they will perform convolutions in the same way producing these feature maps and different image size wont affect how convolutions take place thus this answers why input shape doesn't require weight changes for convolution but if image shape that has been changed and when convolotions are performed on them using tranfer learning models which are trained on different image sizes(Imagenet data) , then you will see that just before fcl's there is a flattening operation which flattens the final feature maps which then act as input for the fcl's now just see imagenet model will produce diffrent size feature maps which when flattened will have diffrent dimension as compared to yours when it will be flattened thus total number of input neurons for the fcl's may get reduced or increased depending upon your image size(feature map size after convolution +pooling) and when you reduce the number of neurons you will need to retrain them to get new weight values and also your output layer is also modified depending upon total number of classes you are predicting hence this ANN that is the fcl's there architecture is modified as per your image size and output classes hence changing the architecture requires changing the weights which is done using training through backpropogation.Thus this concludes image sizes and your problem dont affect the convolution(because weights of filters are updated and image size doesn't affect filter size but only the size of the feature map produced after convolution) process but they do affect the ANN or the fcl's hence weights are updated through training.

P.S- you can actually even update the weights of the convolutions by unfreezing the whole model and then re-training the whole network even the backbone fastai provides a great support for this visit there documentation and site it is quite helpful.

Answer 2

In convolution layers, weights are the filter values.
Weights of a CNN network is determined by the number and size of filters. Weights doesn't depend on shape of input image.
Filters of pre-trained network are capable of detecting different features in an image. Consider sobel filter which detects edges in an image. A 3x3 sobel filter can be used to detect edges in an image of any size. Similarly, those filters can be reused instead of again learning from scratch.

That's why, weights of CNN network can be transferred to another CNN network with different input shape. However, number of layers, no. of filters, filter sizes and number of image channels must be same.

For transfer learning, best practices would be to use pre-trained model for similar task and don't change the input shape to very small or large.

On the other hand, weights of Fully Connected(Dense) layer can't be transferred. Because, those weights depend on image size. Dense layers at the end of CNN network should be cut off and recreated to adopt to change in image size.
It would be more clear if you get to know how the trainable parameters are calculated in CNN and Dense layers. It would be very long if I try to explain in this answer. You can google search on that.