0
$\begingroup$

What is the correct procedure for using a validation set to reduce overfitting?

Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply this model to the validation set then get 20%. What do I do then?

How can the validation set be used to reduce overfitting especially with reference to Naïve Bayes?

$\endgroup$
0
$\begingroup$

It is not much about the algorithm you use. It is about the fact that you learned so much details from your training set so just tune the parameter inside a loop in which training and validation errors are calculated each time. In NB case, you do not have many parameters in that sense. Probably the features can be inspected instead of parameter. Then see the famous point at which the validation error is the minimum.

Do not use fixed splits. For each calculation in loop do $n$ times splitting + evaluation and take the mean and std of errors. Gives you better impression about the stability of the results and effectiveness of your algorithm.

$\endgroup$
  • $\begingroup$ Is there some literature that j can read about your last point? $\endgroup$ – Vykta Wakandigara Nov 7 '18 at 13:00
  • $\begingroup$ This is a general rule in statistical model selection. You either use k-fold cross validation which does it internally or you try your algorithm several times by random sampling just to be sure that the result is not "special" by taking the mean of all errors. see these two stat.cmu.edu/~ryantibs/datamining/lectures/18-val1.pdf and math.canterbury.ac.nz/~r.vale/Crossvalidation.pdf $\endgroup$ – Kasra Manshaei Nov 7 '18 at 13:04
  • $\begingroup$ You can use scikit-learn's GridSearchCV. It'll do the k-fold cross validation for you and help you in tuning the hyperparameters. $\endgroup$ – Yash Jakhotiya Jul 7 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.