Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Let's say that we have a binary classification problem.
Why would it be bad to fit a linear regression and then classify given a threshold?
The output would be continuos and it could be out of range, but we could convert it to binary with the threshold.
It would work, afterall ML is a lot about engineering and hacking the things together. However, it would perform not so well, as for example logistic regression. If you compare the a linear line and logistic regression you will notice that the gradients of their respective loss functions for points near the decision boundary ("threshold") and far away from it are quite different. While for logistic regression points, that lie close to the decision boundary or are wrongly classified the absolute value of the gradient is larger. As a consequence the optimization, if a gradient approach is chosen, will be impacted by these points the most. Thus, the curve will be fitted to improve the classification.
Whereas in linear regression the distance of each point to the linear line is to be minimized, so independently of if the point is already classified correctly, it will impact the optimization. This leads to the optimization to be influenced a great deal by points that are already classified correctly and don't really need to influence the adjustement of the linear function. They suppress the importance of the points that are actually wrongly classified and that should impact the optimization at large.
I'm taking a more "applied" view here:
Normal (OLS) regression is linear and can take on any value for the predicted dependent variable $\hat{y}$.
In contrast, Logit (via the logistic link function) restricts the outcome to $\hat{y} \in [0,1]$.
This is a desireable property as you can interpret the predicted values directly as a probability. In fact, classification is done by splitting predictions at $0.5$.
Here is a little illustration in R:
# Fake Data
df1=data.frame(rep(1,1000),sample(0:30,1000,rep=TRUE))
df2=data.frame(rep(0,1000),sample(20:100,1000,rep=TRUE))
colnames(df1)<-c("y", "x")
colnames(df2)<-c("y", "x")
df=rbind(df1,df2)
# Linear Model
ols = lm(y~x, data=df)
ols_preds=predict(reg,newdata=df)
# Plot
plot(df$x, ols_preds, type="p")
lines(df$x, df$y, type="p",col="red")
# Logit
logit = glm(y ~ x, data = df, family = "binomial")
logit_preds=predict(logit,newdata=df,type = "response")
# Plot
lines(df$x, logit_preds, type="p",col="blue")
Result:
In the plot below,
As you can see, OLS "overshoots". It can predict "probabilities" below 0 or above 1 and it does predict a probability of "below 0" in this example.
You can also see that OLS (assuming we "split" classes at 0.5) does not produce a very good split. The reason is that my data $x$ for class $y=0$ is in a range between 0 and 30, while for $y=1$ it is in a range between 20 and 100. So the "skewed" $x$'s also lead to a bad split under OLS.
Logit in contrast splits at a lower $x$ value compared to OLS, which is desireable since this split comes closer to the true classes.
This ultimately come from the objective function. It is different for Logit compared to OLS (in which simply the sum of squared residuals is minimised). Logit looks at "log odds" that some observation belongs to class $y$. So a very different perspective on $y$.
I think one of the most significant issues is the loss measurement. For a point with true value 0, a predicted value of -1 or 1 contribute the same to the loss, but these are not equally bad predictions!
