Logic behind pre-trained weights and transfer learning

Question

I am not sure about the logic behind, how pre-trained weights actually make sense and translate into a new problem.

To be more specific; for example in a object detection network, how would a model's weights that was trained, let's say, on the COCO dataset, with 80 categories, would translate into my new problem that only has 2 categories (classes). How does this make sense? What kind of meaningful features could even be transfered from the previously pre-trained model to my new problem, since the number of categories (classes) have been changed, and also I'm trying to detect completely different type of objects than the previous model? Why do we do transfer learning then?

Answer 1

If we classify new objects using transfer learning:

1.We delete the top Dense layer of the pre trained neural network.
2.Now suppose you have to classify 5 classes, so your final dense layer will contain 5 nodes.
3.Also you will add some dense layers prior to your new 5 node Dense layer, so that you can train the model with new data.
4.All the layers prior to the few dense layers you added will be frozen.
5.Now you will train your new model with your dataset.

---

The Logic behind these steps are:

1.As CNN uses hierarchical based learning, meaning the initial layers will try to learn basics features in a dataset ie. lines, corners, shapes etc. So the frozen layers of your model have already learnt the basic features which are common to your dataset as well as the dataset it has been previously trained on.
2.The few Dense layers you added, will try to learn high level features specific to your dataset when you train your model with your dataset.

Answer 2

In case of CNN, you are correct in the sense that you cannot use the final layer weights if the number of categories are different. But you CAN reuse the weights in the initial layers. These recognise the lower level objects in the image. There is no need to train all over again. You would only have to train the upper layers specific to the categorisation you want to do. So transfer learning helps for the lower layers in case of CNN.

Let me give one more example. Say there is a trained a model to recognize 100 categories of objects including animals, plants, buildings, vehicles etc. You want to only categorize vehicles. You could use the lower layers of the above model as-is and then add couple of layers at the top which learn the exact types of vehicle categorization.

In case of word embedding, the concept of transfer learning is more intuitive and may help understand the concept better. You can train a model on entire Wikipedia and enable it to learn embeddings for representation of words. These representations can then directly be used in other models - for e.g.. tweet sentiment classification etc. In fact BERT is one such pertained dataset of embeddings. It is a classic case of transfer learning. You no longer have to train on volumes of TeraBytes to learn the language embeddings. You can use it off the shelf.