6
$\begingroup$

Usually the neural network training has at least 2 steps:

  1. first trained on a large set of some standard data (ImageNet, ...)
  2. and then the resulting weights are trained on a small set of my data (in this step we can train all layers or only one last layer)

What is the same of 2-nd step, is it Fine-tuning or Transfer-learning? And what is the different between Fine-tuning and Transfer-learning?

$\endgroup$
1
$\begingroup$

Generally, I would refer to this as transfer learning or network adaptation. That is, taking a network that has learned useful features from one domain and adapting that network and its developed features to another domain.

That said, there appear to be many sources that closely conflate fine tuning with transfer learning. Therefore, I would say the difference in terminology is primarily opinion-based and suggest closure of this question on those grounds.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. Those, is there no significant difference between transfer-learning and fine-tuning? I call can it what I want: transfer-learning - even if I change only last layer (and don't touch other layers). $\endgroup$ – Alex Aug 17 '17 at 20:15
  • 1
    $\begingroup$ You can definitely call it whatever you want. The closer you are to needing other people to be able to completely follow and/or reproduce your results, the more you'll want to specify exactly what it is that you did rather than rely on ambiguous terms, however. $\endgroup$ – Thomas Cleberg Aug 17 '17 at 20:20
1
$\begingroup$

Fine-tuning is the process in which the parameters of a trained model must be adjusted very precisely while we are trying to validate that model taking into account a small data set that does not belong to the train set.

That small validation data set comes from the same distribution as the data set used for the training of the model. The split of the available data to train and validation set is random.


Transfer Learning or Domain Adaptation is related to the difference in the distribution of the train and test set.

So it is something broader than Fine tuning, which means that we know a priori that the train and test come from different distribution and we are trying to tackle this problem with several techniques depending on the kind of difference, instead of just trying to adjust some parameters (usually we are doing this for reasons as preventing overfitting etc.)

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! So, if we have one dataset and randomly separate it to the training (90%) and validation (10%) datasets, then train the model file with training-dataset and after that we train the same model file with validation-dataset - this is Fine-tuning. But if we train the model file with the 1st dataset and after that we train the same model file with the 2nd dataset (with a different distribution of classes/images, or with other classes than in the 1st dataset) - this is Transfer Learning or Domain Adaptation. But do we need to freeze some layers for the Fine-tuning or not? $\endgroup$ – Alex Apr 22 '18 at 10:25
  • $\begingroup$ Yes, you should freeze some layers. You can find an example in the Keras documentation here. $\endgroup$ – Joel Carneiro Jan 14 '19 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.