Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data.
In one of my problems, there are around 150 features and with xgboost I am getting a logloss of around 1 if I use all features. But if I remove around 10 bad features (found using some technique) I am observing a logloss of .45. That is huge improvement.
My question is, can bad features really make such big differences?
Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also has a lot to do with the amount of data points you have compared to the number of features. If you have 10,000,000 data points 150 features is not an issue, if you have 400 data points 150 features is way too much. How can you learn something when there is so little to learn from? Overfitting is a real issue for almost any algorithm, and noisy features make it more easy to learn things that are not actually there.
I am hesitant to think of features as bad features. However, there are features more or less useful relative to other features.
In general pruning features are seen as a good best practice. Overfitting certainly is a real problem within machine learning. Usually a combination of creating too powerful of a model, not having enough data, and/or having too many features can create an undesirable outcome.
Proper partitioning of a dataset can help to adjust the model parameters where warranted helping in a more generalizable model.
