I have built a classification model that I have to refit.

New instances have been added to the former data set on which the model was build.

I was thinking to proceed that way :

  • split my new data set in 3 sub data sets : ( train , test_01 ) data sets to refit the model , ( test_02 ) test data set to compare the original model and the refitted one ( on the metric formerly used ).

Of course the ( train , test_01 ) data sets have to contain the original ones ( those used to fit the former model before being refitted ).

Is this the correct way to proceed?


To make it a fair test, you need to ensure that neither model has seen any of the data in test_02. The closer you can get each subset of the data to match the distribution of the overall combined data set, the better. If you steer by that, the approach you suggested is fine.

A more direct way of answering the question of whether you need a new model is to work out the average accuracy of a new model trained on the updated data set (with the new samples) by using k-fold cross-validation, and then compare that to the accuracy of the average model trained on the old data and tested on the combined data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.