In transfer learning, often only the last layer of the network is retrained using gradient descent. However, the last layer of a common neural network performs only a linear transformation, so why do we use gradient descent and not linear (or logistic) regression to finetune the last layer?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ A neural network with all layers except the final frozen and with the sigmoid activation for the final is the same as logistic regression. Logistic regression is usually optimized by gradient descent. So you seem to juxtaposing two things that are actually the same thing (except in the case you want to later unfreeze the weights of the NN it makes it easier implementation-wise as in the answer). $\endgroup$– ValentasCommented Jan 30, 2022 at 9:35
-
$\begingroup$ @Valentas or, if the NN is doing regression, the final activation function might be linear, so that the transfer learning step of refitting the last layer only is exactly linear regression, with a closed-form solution. $\endgroup$– Ben Reiniger ♦Commented Oct 10 at 13:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
The common approach to fine-tuning an existing pre-trained neural network is the following:
- Given an existing pre-trained neural network model (e.g. imagenet), remove the last layer (which does classification in the pre-training task) and freeze all weights in the remaining layers of the model (usually with setting the trainable parameter to false).
- Add a new final dense layer that is to be trained on the new task.
- Train the model on the new task's dataset until convergence.
- [optional] After fine-tuning is converged, possibly unfreeze the all the layers and train further until convergence with a lower learning rate.
A reason to use gradient descent over a different ML algorithm as you suggest is to enable further training after initial fine-tuning (step #4 above). However, it's not necessary to do this. The approach you suggest (to use take the output of the pre-trained model as input to another ML model) may provide satisfactory performance and be more computationally efficient.
Tradeoffs between these approaches is also discussed in the Keras Transfer Learning guide in the section on the "Typical Transfer Learning Workflow".
-
$\begingroup$ Thanks for the answer. If don't find the discussion in the link. Can you be more specific? Also, what confuses me: When I trained the last layer using linear regression, I still can further train the network using gradient descent. I don't see a problem there. $\endgroup$ Commented Apr 3, 2021 at 7:34
-
-
$\begingroup$ @funkwecker Could you say more about why you would want to use linear regression to calibrate the weights in the last layer of the network and later use gradient descent to fine tune globally? It seems possible to change algorithms for different phases of learning but what are the benefits? $\endgroup$– grovCommented Apr 4, 2021 at 1:21