Gradient boosting vs logistic regression, for boolean features

Question

I have a binary classification task where all of my features are boolean (0 or 1). I have been considering two possible supervised learning algorithms:

• Logistic regression
• Gradient boosting with decision stumps (e.g., xgboost) and cross-entropy loss

If I understand how they work, it seems like these two might be equivalent. Are they in fact equivalent? Are there any reasons to choose one over the other?

---

In particular, here's why I'm thinking they are equivalent. A single gradient boosting decision stump is very simple: it is equivalent to adding a constant $a_i$ if feature $i$ is 1, or adding the constant $b_i$ if feature $i$ is 0. This can be equivalently expressed as $(a_i-b_i)x_i + b_i$, where $x_i$ is the value of feature $i$. Each stump branches on a single feature, so contributes a term of the form $(a_i-b_i)x_i + b_i$ to the total sum. Thus the total sum of the gradient boosted stumps can be expressed in the form

$$S = \sum_{i=1}^n (a_i-b_i) x_i + b_i,$$

or equivalently, in the form

$$S = c_0 + \sum_{i=1}^n c_i x_i.$$

That's exactly the form of a final logit for a logistic regression model. That would suggest to me that fitting a gradient boosting model using the cross-entropy loss (which is equivalent to the logistic loss for binary classification) should be equivalent to fitting a logistic regression model, at least in the case where the number of stumps in gradient boosting is sufficient large.

Answer 1

You are right that the models are equivalent in terms of the functions they can express, so with infinite training data and a function where the input variables don't interact with each other in any way they will both probably asymptotically approach the underlying joint probability distribution. This would definitely not be true if your features were not all binary.

Gradient boosted stumps adds extra machinery that sounds like it is irrelevant to your task. Logistic regression will efficiently compute a maximum likelihood estimate assuming that all the inputs are independent. I would go with logistic regression.