3
$\begingroup$

I was training a binary classifier using XGBClassifier (basically boosted decision trees if I understand it correctly). I have 10K training examples. I have two distinct set of features (but they could be dependent), one contains 26 features (call the set A) and the other contains 96 features (call the set B). I tried to train 3 classifiers, each w/ a different combination of the feature sets, namely A, B and A+B. The result is that using only A is clearly better than using both A and B. At this point I thought it might be overfitting, so that using less number of features actually avoids the overfitting.

The # of trees used in the above trainings was 100. So I used 10-fold cross validation to find the optimal # of trees for each feature set combination, and they are all beyond 100 (like 300-500). So it seems to me the models were learning w/ less-than-desired degree of freedom and they were underfitting if you will.

So this confuses me: why providing additional features make things worse (i.e. using only A is better than using both A and B) when the models are on the under-fitting side (as opposed to overfitting)?

Or maybe a more general question: how do I find out the real problem in this situation?

Note: I have read this question about overfitting/underfitting and I still think my models should be underfitting, because apparently increasing model complexity helps (e.g. from 100 trees to 300-500 trees).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Theoretically speaking, more data leads to better model. However, in practice, more features often leads to the difficulty of model training.

Assuming there are 30 "main" features of your dataset. Feature set A contains 20 "main" features, so it might be easy (20 out of 26) for one "main" feature to be "chosen" and "trained", under certain hyperparameters (in your case, 100 trees). When it comes to feature set B, which contains all "main" features, it's hard (30 out of 96) for one "main" feature to be chosen, and it's harder when there's only 100 trees (cause there are 66 "minor" features which should not be trained, relatively). That's what we called "under-fitting".

Back to your question, when the model is under-fitting, if we're lucky that model trained on feature set B (namely model B) contains all these 30 "main" features, the model will be good, and we might not able to find out it's under-fitting. But in most cases, we're not that lucky, model can be ruined with all 66 "minor" features, relatively.

In my ML practice, I'll try more training iterations when more samples come, and try a more complex model when more features come, with the control of over-fitting/under-fitting.

Improve this answer
$\endgroup$
2
  • $\begingroup$ XGBClassifier is gradient boosting decision trees. I thought it will select the most discriminating feature first, before other unimportant features. So I'd assume that adding more unimportant features shouldn't hurt the learning process (since those unimportant features should not have been selected).This is different from other models, e.g. SVM, where more features could indeed confuse the learning process (but in that case leading to overfitting). Is my understanding incorrect? $\endgroup$
    – Roy
    Commented Feb 27, 2017 at 17:50
  • 1
    $\begingroup$ XGBoost will try to select the most importance features along with learning, that's correct. However, as the importance is learned during the learning process, under-fitting side of model may not fully recognized which feature is more important. $\endgroup$
    – Icyblade
    Commented Feb 28, 2017 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.