Will repeatedly fine-tuning on new data cause overfitting?

Question

I have a binary classification model which I have trained on a training set. On the validation set its accuracy is ~85%. I set up early stopping which ended training when validation loss increased. Let's call this final model modelA.

Because of the nature of the task I can generate as much training data as I want. I have a huge dataset (say, reddit comments) and am generating positive and negative examples in a way that there are an astronomical number of combinations. So I'm not worried that the model is "memorizing" my training data.

So I generated another dataset and fine-tuned modelA until the early stopping trigger was fired. The validation accuracy of this model, modelB, was ~87%.

I repeated the process once more, getting a final validation accuracy of ~89% for modelC.

My question is, will this cause overfitting? Is this iterative fine-tuning a common industry practice?

Answer 1

There could be an overfitting issue indeed. Models shouldn't constantly increase their accuracy with additional trainings, and it could result as a bad generalization, i.e. a failure in classifying new combinations.

I don't know if your model uses neural networks, but if it is so, there are several functions to forget/reset a small part of the trained neurons, so that it avoids overfitting in new trainings. The Dropout function is the most common one, but you could have similar functions in error calculation functions like the Adam Optimization.

If there is no such functions in your model, a solution could to restart from scratch the training including the new data. If the result is similar to model A, it could confirm that model B and C were overfitting.

Finally, overfitting results highly depends on data, Model C could also be correct without overfitting, more details would be necessary to check this out. But generally speaking if you are around 90% or more, you are in an overfitting scenario.