till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data.

In one of my problems, there are around 150 features and with xgboost I am getting a logloss of around 1 if I use all features. But if I remove around 10 bad features (found using some technique) I am observing a logloss of .45. That is huge improvement.

My question is, can bad features really make such big differences?

  • $\begingroup$ How do you calculate the loss? On the training set? $\endgroup$ Feb 27, 2016 at 19:17
  • $\begingroup$ @MartinThoma On the test set (not used to train). $\endgroup$ Feb 27, 2016 at 19:47

2 Answers 2


Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also has a lot to do with the amount of data points you have compared to the number of features. If you have 10,000,000 data points 150 features is not an issue, if you have 400 data points 150 features is way too much. How can you learn something when there is so little to learn from? Overfitting is a real issue for almost any algorithm, and noisy features make it more easy to learn things that are not actually there.

  • $\begingroup$ Adding Jan's great answer. You can make a relative importance plot for the variables in XGBoost using xgb.importance. With that you can a rank of the top X variables. I found that sometimes by limiting the model to the top X, the model improves. $\endgroup$
    – jkyh
    Dec 6, 2016 at 21:55

I am hesitant to think of features as bad features. However, there are features more or less useful relative to other features.

enter image description here

In general pruning features are seen as a good best practice. Overfitting certainly is a real problem within machine learning. Usually a combination of creating too powerful of a model, not having enough data, and/or having too many features can create an undesirable outcome.

Proper partitioning of a dataset can help to adjust the model parameters where warranted helping in a more generalizable model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.