Why am I getting very different results between SVC, LinearSVC and Naive Bayes?

Question

I am doing classification by using bag-of-words model. The goal is to locate users based on their tweets. Splitted the data as 80% training and 20% test.

I did experiments with sklearn's SVC and Naives Bayes. The results 35% and 42% accuracy respectively. However, when I try the sklearn's LinearSVC algorithm, it gives me 80% which is shocking.

This is the part of the code:

text_clf = Pipeline([
    ('vect', CountVectorizer(stop_words='english')),
    ('tfidf', TfidfTransformer()),
    ('clf', LinearSVC()),
])
text_clf.fit(train_data, train_target)  

What might be the reason for that? Why LinearSVC performs really good?

Answer 1

These models are used for convex optimization. which means that there is only one solution for the problem. LinearSVC and SVC are used for same purpose but the optimization techniques used are different. E.g. LinearSVC intercept is penalized while in SVC it isn't. there might be differences scaling or default loss function. Hence they produce different results. LinearSVC tends to optimize faster. in multiclass classification, liblinear does one-vs-rest by default whereas libsvm does one-vs-one.

if you want to know how they can produce similar results. You can check out this question. https://stackoverflow.com/q/33843981/5947203

Now the reason why SVN produced better results than Naive Bayes is that in this problem features are important. Naive Bayes treat them as independent while SVN looks at the interactions between them to a certain degree. Mathematically, One is probabilistic while other is geometrical. The dependencies are not captured by Naive Bayes hence it does not produce good results.