Right way to Fine Tune - Train a fully connected layer as a separate step

Question

I'm using Fine Tuning with caffenet and it works really well but then I read this in Keras blog entry on Fine Tuning (They use a trained VGG16 model):

"in order to perform fine-tuning, all layers should start with properly trained weights: for instance you should not slap a randomly initialized fully-connected network on top of a pre-trained convolutional base. This is because the large gradient updates triggered by the randomly initialized weights would wreck the learned weights in the convolutional base. In our case this is why we first train the top-level classifier, and only then start fine-tuning convolutional weights alongside it."

So as a separate step in Fine tuning they save the output of the last layer before the fully connected layer (the "bottleneck features") and then they train a "small fully-connected model" on those features and only then they put the newly trained fully connected layer on top of the whole net and train the "last convolutional block".

Is this always the right way to do Fine Tuning?

thanks

Answer 1

First of all, you can do this in automatic way by setting the layers to be trainable or not. By selecting this hyper parameter to be false you freez the layer. we know that the first few layers are features extraction and the assumption when you perform fine tuning that the original problem and your problem are correlated, thus the features should be also the same. What is the right thing to do? It is very difficult to find a direct and absolute answer, it depends mainly on the original problem and the new one. In fact, I prefer to fine tune the whole network using different learning rates. You can use smaller learning rate for the convolution layers and larger rate with the fully connected layers.