1
$\begingroup$

Note - I have read this post but still don't understand

I have a Naive Bayes classifier, when I input my training data to test the accuracy, I get 63.05%. When I input my test data, the accuracy is 65.00%.

Why are the training and test accuracy almost identical? For information, my data is split in 70/30. Does this mean that there is no overfitting?

$\endgroup$
2
$\begingroup$

Why are the training and test accuracy almost identical?

Nearly identical performance on the training set and test set is a good outcome, it means the model is doing what it's supposed to do. To give an intuitive comparison:

  • The performance on the training set is equivalent to how well a student can redo the exercises which have been solved by the teacher during class. The student might just have memorized the answers by heart, so it's not a proof that they understand.
  • The performance on the test is equivalent to how well the student can solve some similar exercises that they haven't seen before in a test. This is a much better indication that the student truly understands the topic.

Does this mean that there is no overfitting?

Yes, it proves that there's no overfitting. To keep with my comparison, overfitting is equivalent to memorizing the answers.

However there can be other problems which bias the result:

  • The performance on the test set is 2 points higher than the performance on the test set. This probably means that the test is very small, because if it was a large enough sample the performance wouldn't be higher. If the test set is too small, the performance is less reliable (any statistics obtained on a small sample is less reliable).
  • Accuracy can be a misleading evaluation measure. It only counts the proportion of correct predictions, so if a large proportion of instances belong to the same class then the classifier can just predict any instance as this class and obtain high accuracy. For example here if the majority class is around 63-65%, then it's possible that the classifier didn't learn anything at all. Looking at precision/recall/F1-score gives a more accurate picture of what happens.

[edit] Important note: as Nikos explained in a comment below, my answer assumes that you have a proper test set, i.e. that the train and test sets are sufficiently distinct from each other (otherwise there could be data leakage and the test set performance would be meaningless).

$\endgroup$
4
  • $\begingroup$ true provided train and test samples are sufficiently distinct and do not overlap $\endgroup$
    – Nikos M.
    Feb 27 at 20:16
  • $\begingroup$ @NikosM. I didn't understand your comment? $\endgroup$
    – Erwan
    Feb 27 at 22:30
  • $\begingroup$ I mean that if train and test samples are not sufficiently distinct (meaning test samples represent mostly unseen cases) then high accuracy scores in test may not mean that the model has indeed learned and generalised correctly. The answer is valid given this pre-condition $\endgroup$
    – Nikos M.
    Feb 28 at 11:43
  • $\begingroup$ @NikosM. ok, thanks for the clarification! $\endgroup$
    – Erwan
    Feb 28 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.