2
$\begingroup$

I have a binary classification task where all of my features are boolean (0 or 1). I have been considering two possible supervised learning algorithms:

  • Logistic regression
  • Gradient boosting with decision stumps (e.g., xgboost) and cross-entropy loss

If I understand how they work, it seems like these two might be equivalent. Are they in fact equivalent? Are there any reasons to choose one over the other?


In particular, here's why I'm thinking they are equivalent. A single gradient boosting decision stump is very simple: it is equivalent to adding a constant $a_i$ if feature $i$ is 1, or adding the constant $b_i$ if feature $i$ is 0. This can be equivalently expressed as $(a_i-b_i)x_i + b_i$, where $x_i$ is the value of feature $i$. Each stump branches on a single feature, so contributes a term of the form $(a_i-b_i)x_i + b_i$ to the total sum. Thus the total sum of the gradient boosted stumps can be expressed in the form

$$S = \sum_{i=1}^n (a_i-b_i) x_i + b_i,$$

or equivalently, in the form

$$S = c_0 + \sum_{i=1}^n c_i x_i.$$

That's exactly the form of a final logit for a logistic regression model. That would suggest to me that fitting a gradient boosting model using the cross-entropy loss (which is equivalent to the logistic loss for binary classification) should be equivalent to fitting a logistic regression model, at least in the case where the number of stumps in gradient boosting is sufficient large.

$\endgroup$
  • $\begingroup$ Try both and see how they perform in CV. It's too hard to say which is more appropriate without knowing the data. Performance may be equivalent but they are not identical methods. $\endgroup$ – Hobbes Apr 3 '17 at 17:51
  • $\begingroup$ @Hobbes, see edited question for my reasoning of why they seem like they might be identical. $\endgroup$ – D.W. Apr 3 '17 at 17:58
  • $\begingroup$ Even if they are "equivalent" in the sense above, there will be a difference in how they are trained (e.g. logistic regression will have trouble with collinear features whereas decision stumps don't). $\endgroup$ – oW_ Apr 3 '17 at 20:54
  • $\begingroup$ @oW_, interesting! Can you explain why that is? Why do you say that logistic regression will have trouble with collinear features? My understanding is that you can train a logistic regression model even if there are collinear features; collinearity doesn't prevent training. (It means that the resulting model is not unique, and it might make it harder to interpret the resulting model coefficients in some settings, but it doesn't create trouble for training a classifier.) Why would there be a difference? An explanation of that seems like it might constitute a reasonable answer... $\endgroup$ – D.W. Apr 3 '17 at 21:01
  • $\begingroup$ See, e.g., stats.stackexchange.com/a/260331/2921 $\endgroup$ – D.W. Apr 3 '17 at 21:03
4
$\begingroup$

You are right that the models are equivalent in terms of the functions they can express, so with infinite training data and a function where the input variables don't interact with each other in any way they will both probably asymptotically approach the underlying joint probability distribution. This would definitely not be true if your features were not all binary.

Gradient boosted stumps adds extra machinery that sounds like it is irrelevant to your task. Logistic regression will efficiently compute a maximum likelihood estimate assuming that all the inputs are independent. I would go with logistic regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.