I want to know advantages and disadvantages.Also, What are advantages and disadvantages of transfer leraning?
(Suggestions and edits will be appreciated)
let us discuss advantages of training a deep learning model from scratch:
- Building and training NN from scratch is of a great use in the research field.
- You will know your model to the most basics and can modify it in case needed as per the requirements.
- It will be more efficient in terms of size and training time(because there may be less number of layers as compared to any pretrained model trained for more than one purpose).Best example is VGG16.In case of VGG16 it is trained for 1000 different categories and even if we want only 2 categories we have to train the whole model,because we can't change the layers without deep knowledge about the architecture of VGG16.
- For any beginner(like me) it will be beneficial to start building from scratch and the use the pre-trained model.
- Don't require mastery in Deep Learning to use pretrained models.
- Many pre-trained models are trained for less or mode different purposes,so may not be suitable in some cases.
- It will take large amount of resources(time and computation power) to train big models from scratch.
The advantages of training a deep learning model from scratch and of transfer learning are subjective. It depends a lot on the problem you are trying to solve, the time constraints, the availability of data and the computational resources you have.
Let's consider a scenario, you want to train a deep learning model for a task like sentiment classification based on images of faces. You can
Use a pretrained model: You can use a pretrained model (for example, Resnet-50 or VGG-16) as the backbone for obtaining image features and train a classifier (for example a two layered neural network) on top of it. Here, you keep the backbone part obtained from the pretrained model fixed and only allow the parameters of the classifier to change. This approach is ideal when you want to train a model quickly or without much computational resources. The performance of in this case might not be the best because the pretrained backbone may suffer from domain adaptation.
Train a model from scratch: You can train a deep learning model (for example Resnet-50 or VGG-16) from scratch for your problem. This means that you initialize the model with new parameters i.e, not obtained from a pretrained model. This requires more computational resources (or time) to train but learns all the parameters from the training data. If the training data is sufficiently enough, a model equivalent to the model in approach 1 when trained using this approach should perform better.
Transfer and fine tune: You can initialize a model (say again Resnet-50 or VGG-16) with pretrained parameters and fine tune on your dataset. This approach requires more computational resources than approach 1 and equivalent computational resources to approach 2 for the same model. Transfer learning is advantageous if the source domain (tasks on which the model is pre-trained) is related to the target domain (tasks for which you want to train the model). For example, a classification task on Imagenet is related to sentiment classification on images of faces. This is the most commonly used approach if you want the best performance.
Learn more about transfer learning here.
If you have enough computational resources, transfer a pretrained model from related task and fine tune it.
If you are low on computational resources or time, use a pretrained model as backbone and tune the head for your task.
If you can not find a good source domain for transfer learning or you do not want to pretrain a model or maybe the training data is sufficiently large then you can train the model from scratch without using pretrained parameters.