How would you optimize a pre-trained neural network to apply it to a separate problem? Would you just add more layers to the pre-trained model and test it on your data set?

For example, if the task was to use a CNN to classify wallpaper groups, I'm sure that it wouldn't work to directly classify off a pre-trained network trained on images cats and dogs, even though both are image classifiers.


1 Answer 1


Do Read - When Transfer Learning is Disadvantageous ?(thanks to @media) (looks very informative to me, so added here to make this answer a complete one...)

Answer to your Question..(starts here)

Transfer Learning is What you are Looking for..

  • When we are given a Deep Learning task, say, one that involves training a Convolutional Neural Network (Covnet) on a dataset of images, our first instinct would be to train the network from scratch. However, in practice, deep neural networks like Covnet has a huge number of parameters, often in the range of millions. Training a Covnet on a small dataset (one that is smaller than the number of parameters) greatly affects the Covnet’s ability to generalize, often result in overfitting.

  • Therefore, more often in practice, one would fine-tune existing networks that are trained on a large dataset like the ImageNet (1.2M labeled images) by continue training it (i.e. running back-propagation) on the smaller dataset we have. Provided that our dataset is not drastically different in context to the original dataset (e.g. ImageNet), the pre-trained model will already have learned features that are relevant to our own classification problem.

When to fine tune Models?

  • In general, if our dataset is not drastically different in context from the dataset which the pre-trained model is trained on, we should go for fine-tuning. Pre-trained network on a large and diverse dataset like the ImageNet captures universal features like curves and edges in its early layers, that are relevant and useful to most of the classification problems.

  • Of course, if our dataset represents some very specific domain, say for example, medical images or Chinese handwritten characters, and that no pre-trained networks on such domain can be found, we should then consider training the network from scratch.

  • One other concern is that if our dataset is small, fine-tuning the pre-trained network on a small dataset might lead to overfitting, especially if the last few layers of the network are fully connected layers, as in the case for VGG network. Speaking from my experience, if we have a few thousand raw samples, with the common data augmentation strategies implemented (translation, rotation, flipping, etc), fine-tuning will usually get us a better result.

  • If our dataset is really small, say less than a thousand samples, a better approach is to take the output of the intermediate layer prior to the fully connected layers as features (bottleneck features) and train a linear classifier (e.g. SVM) on top of it. SVM is particularly good at drawing decision boundaries on a small dataset.

Fine-tuning Techniques

Below are some general guidelines for fine-tuning implementation:

  • The common practice is to truncate the last layer (softmax layer) of the pre-trained network and replace it with our new softmax layer that are relevant to our own problem. For example, pre-trained network on ImageNet comes with a softmax layer with 1000 categories.

  • If our task is a classification on 10 categories, the new softmax layer of the network will be of 10 categories instead of 1000 categories. We then run back propagation on the network to fine-tune the pre-trained weights. Make sure cross validation is performed so that the network will be able to generalize well.

  • Use a smaller learning rate to train the network. Since we expect the pre-trained weights to be quite good already as compared to randomly initialized weights, we do not want to distort them too quickly and too much. A common practice is to make the initial learning rate 10 times smaller than the one used for scratch training.

  • It is also a common practice to freeze the weights of the first few layers of the pre-trained network. This is because the first few layers capture universal features like curves and edges that are also relevant to our new problem. We want to keep those weights intact. Instead, we will get the network to focus on learning dataset-specific features in the subsequent layers.

You need to train them again in this case as if I am not wrong then the wallpapers is not a class of Image-net models.. It won't be tough to build a model from scratch to do so(preferably a shallower one will also do here..)

Source of my answer is translation from these amazing courses..

For Further Readings,

  • $\begingroup$ If you're only doing feature extraction, as in not fine tuning any of the layers. Is there a reason to replace the last dense layer with a linear classifier? Why not keep it as a new dense layer, and just retrain while keeping all bottom layers frozen? Is it just because training a linear classifier will be faster than training a new dense layer? $\endgroup$ Mar 12, 2019 at 6:44
  • 1
    $\begingroup$ For feature extraction, no need to append layer's but few layers like Pooling and Shape resizing are appended to match the dims and possibly reduce the intermediate values; Surely it's faster to train a linear model than a NN but you can try both and see what works the best for your use case; Unfreeze the last dense layer and retraining with few additional layers to match your problem at hand is a common practice $\endgroup$
    – Aditya
    Mar 12, 2019 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.