# Why would it be bad to fit a regression model to a binary classification problem?

Let's say that we have a binary classification problem.

Why would it be bad to fit a linear regression and then classify given a threshold?

The output would be continuos and it could be out of range, but we could convert it to binary with the threshold.

It would work, afterall ML is a lot about engineering and hacking the things together. However, it would perform not so well, as for example logistic regression. If you compare the a linear line and logistic regression you will notice that the gradients of their respective loss functions for points near the decision boundary ("threshold") and far away from it are quite different. While for logistic regression points, that lie close to the decision boundary or are wrongly classified the absolute value of the gradient is larger. As a consequence the optimization, if a gradient approach is chosen, will be impacted by these points the most. Thus, the curve will be fitted to improve the classification.

Whereas in linear regression the distance of each point to the linear line is to be minimized, so independently of if the point is already classified correctly, it will impact the optimization. This leads to the optimization to be influenced a great deal by points that are already classified correctly and don't really need to influence the adjustement of the linear function. They suppress the importance of the points that are actually wrongly classified and that should impact the optimization at large.

• Could you elaborate on the gradient bit? For example, a confidently wrong prediction also lands on the low-gradient side of the sigmoid, but the log-loss is extremely large there, right? Commented Jun 1, 2020 at 18:42
• True, it should be the gradients of the loss functions. I changed it accordingly. Hope its clearer now. Commented Jun 1, 2020 at 18:56

I'm taking a more "applied" view here:

Normal (OLS) regression is linear and can take on any value for the predicted dependent variable $$\hat{y}$$.

In contrast, Logit (via the logistic link function) restricts the outcome to $$\hat{y} \in [0,1]$$.

This is a desireable property as you can interpret the predicted values directly as a probability. In fact, classification is done by splitting predictions at $$0.5$$.

Here is a little illustration in R:

# Fake Data
df1=data.frame(rep(1,1000),sample(0:30,1000,rep=TRUE))
df2=data.frame(rep(0,1000),sample(20:100,1000,rep=TRUE))
colnames(df1)<-c("y", "x")
colnames(df2)<-c("y", "x")
df=rbind(df1,df2)

# Linear Model
ols = lm(y~x, data=df)
ols_preds=predict(reg,newdata=df)

# Plot
plot(df$$x, ols_preds, type="p") lines(df$$x, df$y, type="p",col="red") # Logit logit = glm(y ~ x, data = df, family = "binomial") logit_preds=predict(logit,newdata=df,type = "response") # Plot lines(df$x, logit_preds, type="p",col="blue")


Result:

In the plot below,

• red are original data,
• black are predicted values from the linear (OLS) model, and
• blue are the predicted values from the Logit.

As you can see, OLS "overshoots". It can predict "probabilities" below 0 or above 1 and it does predict a probability of "below 0" in this example.

You can also see that OLS (assuming we "split" classes at 0.5) does not produce a very good split. The reason is that my data $$x$$ for class $$y=0$$ is in a range between 0 and 30, while for $$y=1$$ it is in a range between 20 and 100. So the "skewed" $$x$$'s also lead to a bad split under OLS.

Logit in contrast splits at a lower $$x$$ value compared to OLS, which is desireable since this split comes closer to the true classes.

This ultimately come from the objective function. It is different for Logit compared to OLS (in which simply the sum of squared residuals is minimised). Logit looks at "log odds" that some observation belongs to class $$y$$. So a very different perspective on $$y$$.

I think one of the most significant issues is the loss measurement. For a point with true value 0, a predicted value of -1 or 1 contribute the same to the loss, but these are not equally bad predictions!

• I see your point, thanks!! but I believe that there has to be a more solid answer, maybe you have more thoughts Commented Jun 7, 2020 at 11:56
• The last paragraph of the answer stackoverflow.com/a/54458777/10495893 is worth a read. But then the ISL section referenced seems to suggest that (in the two-class case) the linear probability model may not be so bad. (I wish they had a reference for "it can be shown that".) The Andrew Ng video gives an interesting example, that sort of reflects my answer: that new extreme point gets a large loss, but in the "wrong" direction, pulling the model away from what seems reasonable; a new threshold salvages the hard classification, but the instability of the "probabilities" is worrying. Commented Jun 7, 2020 at 13:46
• I quote Andrew NG from the video "usually applying linear regression to a dataset: "you might get lucky but often is not a good idea" Commented Jun 7, 2020 at 16:37