0
$\begingroup$

Just a quick question, I am building a ML model right now however I am receiving very similar (72.2 and 72.4 for example)% for both Accuracy and F1-Score on my Validation Dataset and my unseen Test Set respectively. This is occuring on most of the baseline models I have produced for my problem right now.

Is this showing that my model is completely overfitting or just acting completely random and getting lucky.

Thanks

$\endgroup$
3
  • $\begingroup$ To check if your model is overfitting you will have to compare against the performance on the training set. Do you find performance metrics on the train set significantly higher than the out of sample test sets? $\endgroup$ May 6 at 15:58
  • $\begingroup$ Performance is extremely similar which is worrying me? $\endgroup$ May 6 at 19:52
  • $\begingroup$ Ok, try training the model on a much smaller train set. Check if that increases performance on train set and degrades performance on test set. $\endgroup$ May 7 at 4:01
0
$\begingroup$

If the training set, validation set and the unseen test set (as you put it) have the same score, but lower than you expected then the model has not overfitted.

An overfitted model would have higher scores for the training data at least, and depending on how you optimised the hyper parameters on the validation data but lower for the unseen test set.

The more likely outcome is that your model has underfitted i.e. low but consistent scores across all 3 sets of data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.