1
$\begingroup$

They starts from the same equation as below.

y = w*x + b

But they solve it differently. MLR specified the w and b by minimizing the square error whereas SVM specified w and b by minimizing the loss function defined by C and epsilon.

I am wondering if the result of regression is significantly different. I guess that if the given data set is clean and well-explained by input features, the resultant w and b between SVM and MLR will be close. Putting my original question differently, I don't find any reasons to use linear SVM regression over multiple linear regression.

| improve this question | |
$\endgroup$
0
$\begingroup$

They could potentially be very different, because linear regression penalizes using squared error while SVRs penalize absolute error. You might have learned that "outliers" can have an outsized influence on the regression line because of this--this won't be the case with SVR, however.

Furthermore, if the SVR you're talking about is like the one described here (great resource there, by the way) then choosing a larger $\epsilon$ will yield a flatter regression line/hyperplane, whereas you don't have such a tuning parameter for regular ol' linear regression. This parameter enables you to try to fight overfitting by flattening the regression line.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer! I think the difference is significant only when the weight of outliers or noise is not negligible. (That's why why I said "when the given data set is clean"). but in that case, isn't it much better and robust using noise filters before applying MLR in terms of implementation time&efforts vs performance? Also, is there any way to quantitatively compare the quality of these two regressions? The result reported by CV-RMSE or r2 would be disadvantageous to SVR because it is MLR that is wired in a to minimize such a reported error regardless of the quality of regression. $\endgroup$ – Shlee Aug 29 '19 at 20:15
  • $\begingroup$ Remember, what you really care about in the end is predictiveness on a test set. A noise filter might give you "cleaner" data but might hurt you on test set accuracy! The gold standard of comparison is examining the empirical losses of the models on a test set. If you care more about L1 loss than L2 loss, validate on L1 loss. The loss function of the model is just a means for fitting the model. It's even possible that linear regression will perform worse in test RMSE because it's overfitted! $\endgroup$ – Sheridan Grant Aug 30 '19 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.