1
$\begingroup$

What is the correct procedure for using a validation set to reduce overfitting?

Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply this model to the validation set then get 20%. What do I do then?

How can the validation set be used to reduce overfitting especially with reference to Naïve Bayes?

Improve this question
$\endgroup$
1
  • $\begingroup$ it's not as much about the validation set itself as the model tuning. but anyway, look for cross validation. might help a little. $\endgroup$
    – Reza Keshavarz
    Commented Jul 2, 2020 at 4:54

1 Answer 1

Reset to default
1
$\begingroup$

It is not much about the algorithm you use. It is about the fact that you learned so much details from your training set so just tune the parameter inside a loop in which training and validation errors are calculated each time. In NB case, you do not have many parameters in that sense. Probably the features can be inspected instead of parameter. Then see the famous point at which the validation error is the minimum.

Do not use fixed splits. For each calculation in loop do $n$ times splitting + evaluation and take the mean and std of errors. Gives you better impression about the stability of the results and effectiveness of your algorithm.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Is there some literature that j can read about your last point? $\endgroup$
    – Vykta Wakandigara
    Commented Nov 7, 2018 at 13:00
  • $\begingroup$ This is a general rule in statistical model selection. You either use k-fold cross validation which does it internally or you try your algorithm several times by random sampling just to be sure that the result is not "special" by taking the mean of all errors. see these two stat.cmu.edu/~ryantibs/datamining/lectures/18-val1.pdf and math.canterbury.ac.nz/~r.vale/Crossvalidation.pdf $\endgroup$
    – Kasra Manshaei
    Commented Nov 7, 2018 at 13:04
  • $\begingroup$ You can use scikit-learn's GridSearchCV. It'll do the k-fold cross validation for you and help you in tuning the hyperparameters. $\endgroup$
    – Yash Jakhotiya
    Commented Jul 7, 2019 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.