1
$\begingroup$

I tried to test my model with seen and unseen data (seen data are data that i used to learn the model). I figure out that as much as i increase the number of features seen data can be properly predicted, while when using a feature selection technique unseen data can be properly predicted. Is there any explanation about this.

Thanks in advance

$\endgroup$
1
$\begingroup$

What you want to do in building predictive models , in general , is closing the gap between the loss on seen data ( training data ) and unseen data ( test data ). Increasing the numbers of features will naturally make the hypothesis ( the function that is meant to learn from your training and will receive unseen data ( X ) to predict your target value ( Y ) ) more complex and can fit your training data perfectly at a certain point, BUT there is a thresh-hold ( which you went above according to your results by increasing the number of features) where your model starts giving very good results on your training set and much poorer results on the test set, and that is called overfitting which is a problem caused by high variance in your model which basically means you are allowing your function to be complex enough to fit your training data perfectly, and be less able to predict unseen data ( your test data ).

Go with the feature selection technique if it reduces the gap between your training loss and test loss!

Notice here in this example that increasing the number of features gives lesser test loss but, at a certain point, the loss on test starts going up again.

Image

$\endgroup$
  • $\begingroup$ Thanks for the answer, it's clear now $\endgroup$ – Born New May 10 '19 at 13:43
  • $\begingroup$ If my answer was clear, please mark it as accepted! $\endgroup$ – Blenz May 10 '19 at 13:47
  • $\begingroup$ @Blenzus It's generally advisable to wait a day before accepting an answer to encourage other responses; people are less likely to post their own answer to a question that already has an accepted answer. Don't get me wrong, this is a fine answer, but it's not making full use of the format to immediately accept an answer posted within 15 minutes of the question. $\endgroup$ – Nuclear Hoagie May 10 '19 at 14:09
  • $\begingroup$ yes you're right. I didn't see when the question was asked. But i asked for my answer to be accepted since he said it cleared it up for him. So no biggie. $\endgroup$ – Blenz May 10 '19 at 14:12
1
$\begingroup$

For data with lots of features, it's generally the case that many of those features will be unrelated or weakly related to your target variable of interest. It's possible to build a model using even these uninformative features, and you can oftentimes find a pattern in a specific set of samples - with enough features, it's likely that a even a noisy feature appears to be informative in a sample subset (your training data). The problem is that these features are not informative in the general sense, so a model built using these features will perform well on your training data but poorly on unseen test data. This is called overfitting, and it means that your model is too specific to the training data, and does not generalize well.

Feature selection can help to eliminate these irrelevant features before the model-building process, which can greatly improve the performance of many algorithms. You typically want to see similar performance on both the training and test data, which indicates that your model building process performs equally well on both seen and unseen data.

One thing to note, be sure to do your feature selection using the training data only, do not use the full dataset to do feature selection and then split into a train and test set for model building and evaluation. If you do that, you will have tainted your test set with your feature selection, making your test metrics biased and very likely overoptimistic.

$\endgroup$
  • $\begingroup$ Ok, your answer is informative, thanks. $\endgroup$ – Born New May 10 '19 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.