0
$\begingroup$

To test the performance of my model based on some selected features, i try to use unseen and seen data. However, when choosing the accuracy based on all data, the model is almost overfitting since the size of seen data is larger than the other. When i take only the unseen data. The choosen features do not guarantee that the seen data are properly predicted. Is there any way to surpass this issue.

Thanks

$\endgroup$
2
$\begingroup$

The test of your model should always be done exclusively on unseen data. That is fundamental to asses your model's capacity to generalize and predict observations it has never seen. If you test it using already seen data, that is like cheating, and overfitting will be 100% guaranteed.

Repeat the test only on unseen data, and check the difference between train and test performances. If differences are big, then you have an overfitting problem. There are several ways to fight overfitting, but they depend on the nature of your model. Could you please provide some information on it?

$\endgroup$
  • $\begingroup$ thanks for the response, it's a linear SVM $\endgroup$ – Born New Jun 8 '19 at 11:09
  • $\begingroup$ If you can, use an RBF kernel for SVM. They are more powerful models. To fight overfitting, I suggest you to use ensemble of models. Combine its prediction with the predictions form other ML models (Random Forests, kNN, naive Bayes). Typically, a set of weak learners can be combined into a strong learner. $\endgroup$ – Leevo Jun 8 '19 at 11:12
  • $\begingroup$ Ok thanks, combining predictions ( means boosting algorithms?) $\endgroup$ – Born New Jun 8 '19 at 11:23
  • $\begingroup$ Combining multiple ML models is called ensemble learning. There are many ways you can build ensembles. Boosting is one of them, and it's a very powerful tool. There are many sources, this one is a good start: towardsdatascience.com/… $\endgroup$ – Leevo Jun 8 '19 at 12:27
  • $\begingroup$ Neural Networks have their own ways of reducing overfitting, such as dropout and other regularization techniques, let me know if you work with them. $\endgroup$ – Leevo Jun 8 '19 at 12:28
0
$\begingroup$

You can try a leave-one-out cross validation (LOOCV) scheme. If you have $n$ data points, create $n-1$ models. For each model, the training set the entire data set minus one point, the testing set only being that one point, for each point. You can average the error on the unseen point as an unbiased estimate.

I would also look into Random Forests, they will allow you to combine the entire data set and use bagging to find estimates of performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.