Why is the accuracy of a LinearSVC not the same as the SDGClassifier?

Question

I'm fine tuning parameters for a linear support vector machine. There are multiple ways to do it, but I wanted to compare LinearSVC and SDGClassifier in terms of time. I expected the accuracy score to be the same but, even after fine tuning with GridSearchCV, the score of the LinearSVC is lower. I tried changing up parameters many times, but the maximum with LinearSVC I can get is 41.176 versus 41.503 of SDGClassifier. Why?

The code:

    class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import SVC, LinearSVC
    from sklearn.pipeline import Pipeline


    self.pipeline = Pipeline(
        [
            ('vect', CountVectorizer()),
            ('tfidf', TfidfTransformer()),
            #('tfidf', TfidfVectorizer()),
            ('clf', LinearSVC( loss='hinge',
              penalty='l2', max_iter = 10,



             #SGDClassifier(
                #loss='hinge',
               # penalty='l2',
              #alpha=1e-3,
              #  random_state=42,
             #max_iter=100,
                #learning_rate = 'optimal',
                #tol=None



            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    from sklearn.model_selection import GridSearchCV
    from sklearn.svm import SVC
    from sklearn.linear_model import SGDClassifier
    from sklearn import svm
    from sklearn import preprocessing
    from sklearn.preprocessing import LabelEncoder, OneHotEncoder
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.svm import LinearSVC
    train_df = self.read_data(train_file, lower_case)
    param_range = [0.001, 0.01, 0.1, 1, 10, 100]
    parameters = {
        # 'vect__ngram_range': [(1, 1), (1, 2)],
        'tfidf__use_idf': (True, False),
        #'clf__alpha': [0.0001, 0.001, 0.01, 1, 10, 100],
        'clf__max_iter': [10, 100, 1000],
        'clf__tol': [0, 0.0001, 0.001, 0.01],
        'clf__loss':['hinge'],
        'clf__penalty': ['l2'],
        'clf__C': param_range

    }


    parameters1 = {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['linear'], 'clf__tol' : [0, 0.01]

                  }
    lr = LinearSVC()
    print(lr.get_params().keys())
    gs_clf = GridSearchCV(self.pipeline, parameters, cv=5, n_jobs=-1)
    gs_clf = gs_clf.fit(train_df['text'], train_df['truth'])
    print(gs_clf.best_score_)
    for param_name in sorted(parameters.keys()):
        print("%s: %r" % (param_name, gs_clf.best_params_[param_name]))

    # estimator_svm.best_score

    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)

    test_df['pred'] = learner.predict(test_df['text'])
    return test_df

Answer 1

You are implementing the same model but with different optimization techniques, which means the final weights will be different but if the problem is convex it should converge to the same value.

• SGD is dependent on initialization (as most iterative processes);
• The convergence might take way too long is learning rates are not set properly;
• Since SGD uses batches instead of optimizing on the entire dataset at once, it might not reach global optimum for the whole training dataset.

SGD might give better final scores since:

• Your final scoring metric optimum might differ from loss function optimum;
• Test/Validation optimum will probably differ from training optimum.

Also, as pointed by N. Kiefer it is worth to check if the 0.4 points difference on scores are really statistically significant.

Answer 2

When used with loss="hinge" The SGDClassifier gives a LinearSVM, so they should be the same. This is matter of choosing the same hyperparameters for both. Can you check that you using the exact same parameters?

As a side note (I don't know your dataset), 41.1 and 41.5 looks pretty similar, this also might be about splitting the training/testing data exactly the same and stopping the training at the correct time.

Answer 3

Remember that

LinearSVM uses the full data and solve a convex optimization problem with respect to these data points.

SGDClassifier can treat the data in batches and performs a gradient descent aiming to minimize expected loss with respect to the sample distribution, assuming that the examples are iid samples of that distribution.

As a working example check the following and consider:

1. Increasing the number of iterations
2. Increase the number of iterations before the early stoping
3. Both classifier should be using the same loss function, in this case "squared hinge"

from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y= True)
X_train ,X_test, y_train, y_test = train_test_split(X, y, test_size = .3, random_state = 42)

sgd = SGDClassifier(random_state= 42,loss = "squared_hinge", max_iter= 100000,n_iter_no_change=1000).fit(X_train, y_train)
linearsvm = LinearSVC(random_state= 42,max_iter= 100000).fit(X_train, y_train)

sgd.score(X_test, y_test)
linearsvm.score(X_test,y_test)

Before modifying these two parameters, SGDClassifier gave ~ 20% less accuracy After both performed the same on the test set