3
$\begingroup$

At the worst case scenario, we could treat the pretrained weights as a random initialization, same as what we would do for training from scratch, right? If that is the case, then wouldn't it be better to always start with a pretrained model, as the lower layers of weights has already probably learnt general patterns of images that are transferable across all data sets?

My worry is what if the dataset I want to use for finetuning is highly specialized, highly unnatural and very different from the dataset the pretrained model is trained on. Would this still mean finetuning from a pretrained model wouldn't be the best idea? (e.g. training on X-Ray images instead of natural images for a cifar-1000 pretrained model.)

Improve this question
$\endgroup$
5
  • $\begingroup$ Please take a look at this $\endgroup$
    – enterML
    Nov 27, 2016 at 15:41
  • $\begingroup$ Yes I get that if the task is similar then fine tuning is good. But how does one define similar? And how does one weigh the benefits of the pretrained architecture having the general ability to look across different image types? I'm afraid I may be venturing into uncharted areas here, since the understanding of what the layers represent is still unclear at the moment. $\endgroup$
    – lee kwot sin
    Nov 27, 2016 at 15:45
  • $\begingroup$ Similarity here means if the task is exactly same or a subset of the problems solved by the pretrained network. The benefit comes in term of the weights learned by the pretrainedd network. They can be directly used for your task, provided the task is similar. $\endgroup$
    – enterML
    Nov 27, 2016 at 16:00
  • $\begingroup$ If the task happens to be quite dissimilar, then are there any disadvantage or advantage in using weights from a pretrained model versus a random initialization? $\endgroup$
    – lee kwot sin
    Nov 28, 2016 at 8:03
  • $\begingroup$ Andrew Ng gives the example of using a network trained on cats to for radiology examples. The lower layers (edges, corners, shapes) can add some benefit. $\endgroup$
    – Tom Hale
    May 30, 2018 at 12:49

1 Answer 1

Reset to default
0
$\begingroup$

One limitation of using a pretrained model is that you are forced to use the same architecture and weights. There could many scenarios where the pretrained architecture is limiting. If you train from scratch, you can define a custom architecture for the specific problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.