Recently I came across this model called RidgeClassifier, It coverts the predicted value (y) to {-1,1} and then uses the Ridge Regression. During prediction if the value of y is < 0 then it is predicted as -1 and if the value of y is > 0 then it is predicted as +1.

So, when RidgeClassifier works well for pretty well for some datasets, Why dont/cant we have something called LinearClassifer which can use LinearRegression during training and the same model for prediction?

  • $\begingroup$ In other words, why can't we use a regression model for a classification task, by just pretending the dependent variable is continuous? $\endgroup$ Nov 19 at 11:21
  • $\begingroup$ What stops you from doing this method without a ridge regression penalty? $\endgroup$
    – Dave
    Nov 19 at 11:23
  • $\begingroup$ I think this why no classification with regression? explains that....? Why cant we make the same argument for RidgeClassifier, expecting it does not work, But then how does it work @Dave $\endgroup$
    – Vikram
    Nov 19 at 12:57

My stance aligns with what Frank Harrell wrote in the link you gave, so I will not defend the methodology you propose; I am a fan of working with probability outputs. However, nothing stops you from doing the modeling with an OLS regression. After all, the parameters in ridge regression are (typically) calculated as $\hat\beta_{ridge} = (X^TX +\lambda I_{p\times p})^{-1}X^Ty$. If you do the special case of this where $\lambda = 0$, you get the usual OLS equation.

If you are determined to predict on a continuum and then convert to discrete classes, linear regression is just as viable as ridge regression.

Example in R

N <- 100
x <- runif(N,-2, 2)
z <- x
pr <- 1/(1 + exp(-z))
y <- rbinom(N, 1, pr) 
y_alt <- 2*y - 1 # convert to +/- 1
L <- lm(y_alt ~ x)
preds <- predict(L)

Now take preds > 0 as y_alt = +1 and preds < 0 as y_alt = -1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.