How train a pre-trained model based on new dataset?

Question

I have trained a deep nn model based on some existing data. In the meantime, I have collected more data and label them so that I can feed it to the model to improve its performance. The questions is, should I feed:

Option 1- New data to the already trained model?

Option 2- New data to a new model with initialized weights?

Option 3- The entire data (old+new) to a model with initialized weights?

Which method should I choose? Do I have to use only the new data or I have to combine the new data to the old dataset.

I asked this question so that I can choose the option which can improve the overall accuracy and to consider the data drift in the new data.

Answer 1

You need to consider few things while trying to train a pre-trained model over a new set of data.

1. New data may or may not represent the data which the model is initially trained upon. There might be some data drift. So it is important to include the previously used data along with the used data to retrain in-order to make proper predictions. Using only the new data might result in Overfitting or bias. If the new data is large enough and well balanced with less data drift, then it is ok to use only the new data

2. About the pre-trained weights. Your weights will eventually change once you start re-training the model. So re-training your already trained model or using weights from your previous mode onto a new one is basically the same thing but it is much more efficient than building a new model from scratch with no pre-trained weights.

3. Data drift. If the data drift is too significant and it replaces the actual purpose of your previous model then train the model from scratch. If you are trying to achieve something from entirely new data which cannot be done with your old data, you can train from scratch otherwise you can use the weights/re-train the model

Also don't forget that the quality of data must be in par with the old data and there should be no structural changes to it.

If you want to know more about data drift, check out this post