0
$\begingroup$

I have a small restricted dataset, it is not very small but accuracy will be much better if I will have more data. I have split it to train and test datasets: 85%/15%. I have chosen NN model and trained it, measured loss, it is ok. Now I should use it in production. Should I train it additionally on the test data, because in such way I will get better accuracy?

$\endgroup$
  • $\begingroup$ A big no... That way you will not be doing justice to the model as well as to the customers.. $\endgroup$ – Aditya Mar 10 '18 at 17:03
  • $\begingroup$ I would suggest to use Cross Validation first to check which model is actually giving better results. If training all data to deploy in production gives better result, you are good to go. $\endgroup$ – Ankit Seth Mar 12 '18 at 12:44
2
$\begingroup$

The test set is there to show you how the model can perform to unknown cases and basically reassures that it is not overfitted. The selection of the "optimal" model needs to be made based on its performance on the test set and NOT on the training set, because what you need is the model with the best behavior in unknown cases.

In my humble opinion, just splitting the file in training and test set is not the best approach, as there might be bias in the accuracy calculation just because of the random way data were splitted. I would go with cross-validation in order to ensure that the model has been extensively tested. In cases with small amount of data, leave-one-out-cross-validation is often used in order to maximize the amount of available data for the training. You can read more on Cross validation and Leave-one-out-cross-validation (theory and python implementation) here.

When you have found the model with the best performance using cross-validation, I believe you can feel free to train it on the whole set of data before giving it to production.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Great and concise answer! I would also add that in certain cases, it might be a good idea to use stratified sampling you suspect that your data has class imbalance. $\endgroup$ – The Lyrist Jun 11 '18 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.