# Gradient boosting vs logistic regression, for boolean features

I have a binary classification task where all of my features are boolean (0 or 1). I have been considering two possible supervised learning algorithms:

• Logistic regression
• Gradient boosting with decision stumps (e.g., xgboost) and cross-entropy loss

If I understand how they work, it seems like these two might be equivalent. Are they in fact equivalent? Are there any reasons to choose one over the other?

In particular, here's why I'm thinking they are equivalent. A single gradient boosting decision stump is very simple: it is equivalent to adding a constant $a_i$ if feature $i$ is 1, or adding the constant $b_i$ if feature $i$ is 0. This can be equivalently expressed as $(a_i-b_i)x_i + b_i$, where $x_i$ is the value of feature $i$. Each stump branches on a single feature, so contributes a term of the form $(a_i-b_i)x_i + b_i$ to the total sum. Thus the total sum of the gradient boosted stumps can be expressed in the form

$$S = \sum_{i=1}^n (a_i-b_i) x_i + b_i,$$

or equivalently, in the form

$$S = c_0 + \sum_{i=1}^n c_i x_i.$$

That's exactly the form of a final logit for a logistic regression model. That would suggest to me that fitting a gradient boosting model using the cross-entropy loss (which is equivalent to the logistic loss for binary classification) should be equivalent to fitting a logistic regression model, at least in the case where the number of stumps in gradient boosting is sufficient large.

• Try both and see how they perform in CV. It's too hard to say which is more appropriate without knowing the data. Performance may be equivalent but they are not identical methods. – Hobbes Apr 3 '17 at 17:51
• @Hobbes, see edited question for my reasoning of why they seem like they might be identical. – D.W. Apr 3 '17 at 17:58
• Even if they are "equivalent" in the sense above, there will be a difference in how they are trained (e.g. logistic regression will have trouble with collinear features whereas decision stumps don't). – oW_ Apr 3 '17 at 20:54
• @oW_, interesting! Can you explain why that is? Why do you say that logistic regression will have trouble with collinear features? My understanding is that you can train a logistic regression model even if there are collinear features; collinearity doesn't prevent training. (It means that the resulting model is not unique, and it might make it harder to interpret the resulting model coefficients in some settings, but it doesn't create trouble for training a classifier.) Why would there be a difference? An explanation of that seems like it might constitute a reasonable answer... – D.W. Apr 3 '17 at 21:01
• See, e.g., stats.stackexchange.com/a/260331/2921 – D.W. Apr 3 '17 at 21:03