How should I approach training a convolutional neural network in production on new data when I detect model performance degradation due to data or concept drift? Resources like this one and this one lead me to conclude that I need to fine tune the network (as opposed to using transfer learning). Is my conclusion correct? To combat catastrophic forgetting, should I be mixing old data with new data when fine tuning? If yes, any guidelines on how much of the old data I should use? Thanks.
1 Answer
This is a GOOD question. Say you have taken a pre-trained CNN model and then fine-tuned it on your data. The model is productionized and works well for a few weeks but now you observe a drift. The ideal option is to fine-tune with the (old+incremental data) since you anyway used pre-trained models to start out with. The fine-tuning is usually a much shorter process and can be repeated again and again with new data.
You could also look at the option of down-sampling old data and giving more importance to new data that has come in in past few weeks and fine-tune on this combination.
You need not worry of catastrophic forgetting when you are fine-tuning data this way. To be safer, you can freeze most of the layers of the original network and only fine-tune the outermost layers. This way you are not tampering too much with the model and minimally tuning it.
The articles you are referring to are not the right ones in the current situation. You should look at industry papers on incremental learning for CNN models and run thru' some of those for more detailed analysis. For eg:
https://www.mdpi.com/2079-9292/10/16/1879
https://arxiv.org/abs/1712.02719
https://pdfs.semanticscholar.org/8f21/c99d8257c79baf22c211ed17a2224574b524.pdf
Based on the actual situation at ground the strategies need to be chosen according. To repeat, if you are merely using a pre-trained model and fine-tuning it with your data which is what most practitioners do, then it is always better to just fine-tune the entire thing again instead of taking an incremental approach as fine-tuning in general does not take too much time. You can downsample the old data in case the new samples are limited or look at ways to increase the gradient update weightage for the new samples.
-
$\begingroup$ Thanks for this answer. One question on one of the papers you shared, the one titled "Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing." Do you know how the "combined classification" is done when there are multiple branches in the output? The probabilities in each individual branch will add up to 1, but how do you decide which class the input belongs to when one branch predicts between classes c1-c50 and another predicts between classes c51-c60? $\endgroup$ Nov 25, 2022 at 20:38