I'm wondering what steps do you take to decide on the part of the model to unfreeze. Do you do multiple experiments? Since the use of GPU is expensive, you must have some guidelines.
Note: I know the relationship between size of dataset, how close dataset is to the original dataset and how much that impacts whether or not we train more layers. However is there a rule of thumb involving the depth of the model to get the approximate layer?
Example: Try unfreezing model starting from Layer number 169, or layers between 70-100
How much does one need to know specifics of the pretrained model? Can I use it without knowing the architecture?
Thank you for your help!
Regarding #1: As you mentioned, the following amount of fitting you do on a pre-trained (PT) network depends on the relative size and similarity of your data and the data used to train the PT model.
- If our dataset is smaller than the PT data, we should unfreeze fewer layers to avoid overfitting.
- If our dataset is of a similar size as the PT data, we can unfreeze more layers since overfitting is less of a concern.
Essentially, more frozen layers means less overfitting, since those layers will retain the relatively generic features of the PT network. As such, you can use hyperparameter tuning (and the results of your cross-validation) to optimize this like you would any other hyperparameter (grid search, etc).
Regarding #2: Yes, you certainly can use the PT model to fit and make predictions. But, the more you understand the model's architecture, the better you will be able to tune the model, and ultimately, improve your results.