2
$\begingroup$

l have a dataset of dim=(200,2000) 200 examples and 2000 features. l have 10 classes.

l used sklearn for both cases :

svm.svc(kernel=linear)
LinearSVC()

However LinearSVC() performs drastically better than svm with linear kernel. 60% against 23%. l'm supposed to get the same or comparable results since they are fed with same parameters and data.

What's wrong ?

Thank you

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Here is just a guess, but according to me, the linearSVC might perfoms better than SVM with linear kernel because of regularization.

Because linearSVC is based on liblinear rather than libsvm, it has more flexibility and it gives you the possibility to use regularization with your SVM (default is L2-Ridge regularization).

Because you have more features than observations, it exists multiple solutions to your classification model. Some of them are more "robust" than other ones. L2 regularization will help you reduce the ampltitude of your model coefficients and maybe lead to a more stable solution.

More on LinearSVC in the documentation :

http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html

More on Ridge regression benefits when working with a lot of features (and so multicollinearity) :

https://stats.stackexchange.com/questions/118712/why-does-ridge-estimate-become-better-than-ols-by-adding-a-constant-to-the-diago

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer. 1) l still don't understand why accuracies are different . 2) What is the regularization used in svm.SVC() ? do you mean that LinearSVC() gives better accuracy thank to l2-Ridge regularization ? from that l can deduce, we get rid of libsvm and use liblinear for linear problem. l don't think so, l think that we have to get comparable accuracies either with linearSVC() or svm.SVC() since the kernel is linear $\endgroup$
    – Joseph
    Commented Dec 6, 2017 at 16:49
  • $\begingroup$ 1/ Adding regularization, you avoid overfitting when working with too many features and so multicollinearity. Regularization leads to a more robust model. Model is less sensitive to small changes in your features. I recommand you to read more about regularization. This is a primary concern when working with a lot of classifiers and regressors. 2/ There is no regularization with svm.SVC() because this algorithm is based on libsvm() and is more kernel-trick oriented. This is not the same solving. This is why you have to prefer linearSVC() when using linear kernel (and regularization). $\endgroup$
    – Theudbald
    Commented Dec 7, 2017 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.