For example, I have layers that are pretrained. But while predicted, the loss is very high. But not because of pre-trained layers. Because of not pretrained layers. Will every layer be affected by backprop the same?
1
-
$\begingroup$ Not sure I get the question, you will need to freeze these layers as you train. $\endgroup$– GooJCommented Sep 5, 2022 at 22:24
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This depends on how you configure the training process:
You can, for instance, freeze the pretrained layers; this implies that only the not pretrained layers will be updated.
You can also set different learning rates to different layers, so that the pretrained layers are assigned a very small learning rate that allows them to be updated but not too fast.
Therefore, backpropagation is the same for all layers but the weight update strategy can be different.
