# Why would removing a variable in adaboost decrease error rate?

I was trying to classify an outcome on some data using adaboost (the ada package in R) and I was playing around with the training data set of descriptors when I realized that removing a column in the descriptor matrix increased the accuracy of the output on the training data. Specifically, the number of false negatives dropped/true positives increased.

Aside from removing a single column in the descriptors, I left everything else the same, including number of iterations.

Imagine that one of the column is just random data -- then it's not informative at all, so no classifier will be improved by including it.

However, ada's stochastic boosting implementations will always have some chance of including that variable in the classifier it generates. As a result, removing it has the potential to improve the classifiers generated.

(In your case, you might check whether that variable is part of the final model generated.)

• I was under the impression that at every iteration the algorithm would choose the best weak classifier it has available. I guess I've only learned the simple adaboost algorithm and not the stochastic version? Is that where I'm missing knowledge? – amustafa Oct 18 '15 at 14:49
• Ada implements stochastic boosting. Even if you aren't using it, consider this possibility: you have a huge number of columns of random data. At some point in the boosting process, one of those columns is, by chance, a perfect fit for the margins. But of course, it's useless once you're validating the classifier on other data. This sort of behavior is also possible even if you are using a deterministic algorithm. – cohoz Oct 18 '15 at 14:54
• I see, I looked up stochastic methods. Makes a lot more sense now. The deterministic piece also makes sense. Thank you! – amustafa Oct 19 '15 at 4:13

In AdaBoost I guess at each step the model is "greedily" updated and given that typically you regularize heavily and there are bounds on model complexity this can have unexpected results.

For the deterministic AdaBoost I think adding a feature f1 which is random can't degrade performance if the tree depth has no bounds (so the tree can become as complex as it needs to be-remember we are talking of minimizing the training error so overfitting is ok).

In actual implementations however the tree depth has an upper bound which might lead to premature pruning of the tree and hence you might end up getting a higher training error.