Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I tried to test my model with seen and unseen data (seen data are data that i used to learn the model). I figure out that as much as i increase the number of features seen data can be properly predicted, while when using a feature selection technique unseen data can be properly predicted. Is there any explanation about this.
Thanks in advance
What you want to do in building predictive models , in general , is closing the gap between the loss on seen data ( training data ) and unseen data ( test data ). Increasing the numbers of features will naturally make the hypothesis ( the function that is meant to learn from your training and will receive unseen data ( X ) to predict your target value ( Y ) ) more complex and can fit your training data perfectly at a certain point, BUT there is a thresh-hold ( which you went above according to your results by increasing the number of features) where your model starts giving very good results on your training set and much poorer results on the test set, and that is called overfitting which is a problem caused by high variance in your model which basically means you are allowing your function to be complex enough to fit your training data perfectly, and be less able to predict unseen data ( your test data ).
Go with the feature selection technique if it reduces the gap between your training loss and test loss!
Notice here in this example that increasing the number of features gives lesser test loss but, at a certain point, the loss on test starts going up again.
For data with lots of features, it's generally the case that many of those features will be unrelated or weakly related to your target variable of interest. It's possible to build a model using even these uninformative features, and you can oftentimes find a pattern in a specific set of samples - with enough features, it's likely that a even a noisy feature appears to be informative in a sample subset (your training data). The problem is that these features are not informative in the general sense, so a model built using these features will perform well on your training data but poorly on unseen test data. This is called overfitting, and it means that your model is too specific to the training data, and does not generalize well.
Feature selection can help to eliminate these irrelevant features before the model-building process, which can greatly improve the performance of many algorithms. You typically want to see similar performance on both the training and test data, which indicates that your model building process performs equally well on both seen and unseen data.
One thing to note, be sure to do your feature selection using the training data only, do not use the full dataset to do feature selection and then split into a train and test set for model building and evaluation. If you do that, you will have tainted your test set with your feature selection, making your test metrics biased and very likely overoptimistic.
