4
$\begingroup$
  1. I'm wondering what steps do you take to decide on the part of the model to unfreeze. Do you do multiple experiments? Since the use of GPU is expensive, you must have some guidelines.

    Note: I know the relationship between size of dataset, how close dataset is to the original dataset and how much that impacts whether or not we train more layers. However is there a rule of thumb involving the depth of the model to get the approximate layer?

    Example: Try unfreezing model starting from Layer number 169, or layers between 70-100

  2. How much does one need to know specifics of the pretrained model? Can I use it without knowing the architecture?

    Thank you for your help!

$\endgroup$

2 Answers 2

0
$\begingroup$

Regarding #1: As you mentioned, the following amount of fitting you do on a pre-trained (PT) network depends on the relative size and similarity of your data and the data used to train the PT model.

  • If our dataset is smaller than the PT data, we should unfreeze fewer layers to avoid overfitting.
  • If our dataset is of a similar size as the PT data, we can unfreeze more layers since overfitting is less of a concern.

Essentially, more frozen layers means less overfitting, since those layers will retain the relatively generic features of the PT network. As such, you can use hyperparameter tuning (and the results of your cross-validation) to optimize this like you would any other hyperparameter (grid search, etc).

Regarding #2: Yes, you certainly can use the PT model to fit and make predictions. But, the more you understand the model's architecture, the better you will be able to tune the model, and ultimately, improve your results.

$\endgroup$
0
$\begingroup$

The first step to unfreeze the topmost layer. This allows the model to learn the specific targets for the current dataset.

Then layers should be progressively unfrozen. Generally, the higher layers learn non-linear combinations of the lower layers. Lower layers learn task-specific feature selection. It is a trade-off better longer training time and learning better task-specific features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.