0
$\begingroup$

I'm fine tuning parameters for a linear support vector machine. There are multiple ways to do it, but I wanted to compare LinearSVC and SDGClassifier in terms of time. I expected the accuracy score to be the same but, even after fine tuning with GridSearchCV, the score of the LinearSVC is lower. I tried changing up parameters many times, but the maximum with LinearSVC I can get is 41.176 versus 41.503 of SDGClassifier. Why?

The code:

    class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import SVC, LinearSVC
    from sklearn.pipeline import Pipeline


    self.pipeline = Pipeline(
        [
            ('vect', CountVectorizer()),
            ('tfidf', TfidfTransformer()),
            #('tfidf', TfidfVectorizer()),
            ('clf', LinearSVC( loss='hinge',
              penalty='l2', max_iter = 10,



             #SGDClassifier(
                #loss='hinge',
               # penalty='l2',
              #alpha=1e-3,
              #  random_state=42,
             #max_iter=100,
                #learning_rate = 'optimal',
                #tol=None



            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    from sklearn.model_selection import GridSearchCV
    from sklearn.svm import SVC
    from sklearn.linear_model import SGDClassifier
    from sklearn import svm
    from sklearn import preprocessing
    from sklearn.preprocessing import LabelEncoder, OneHotEncoder
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.svm import LinearSVC
    train_df = self.read_data(train_file, lower_case)
    param_range = [0.001, 0.01, 0.1, 1, 10, 100]
    parameters = {
        # 'vect__ngram_range': [(1, 1), (1, 2)],
        'tfidf__use_idf': (True, False),
        #'clf__alpha': [0.0001, 0.001, 0.01, 1, 10, 100],
        'clf__max_iter': [10, 100, 1000],
        'clf__tol': [0, 0.0001, 0.001, 0.01],
        'clf__loss':['hinge'],
        'clf__penalty': ['l2'],
        'clf__C': param_range

    }


    parameters1 = {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['linear'], 'clf__tol' : [0, 0.01]

                  }
    lr = LinearSVC()
    print(lr.get_params().keys())
    gs_clf = GridSearchCV(self.pipeline, parameters, cv=5, n_jobs=-1)
    gs_clf = gs_clf.fit(train_df['text'], train_df['truth'])
    print(gs_clf.best_score_)
    for param_name in sorted(parameters.keys()):
        print("%s: %r" % (param_name, gs_clf.best_params_[param_name]))

    # estimator_svm.best_score

    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)

    test_df['pred'] = learner.predict(test_df['text'])
    return test_df
$\endgroup$
2
  • $\begingroup$ Are the scores you're reporting the grid search's best_score_ (and so the averaged k-fold cross-val score)? You're using potentially a different cv-split each time, so some variation is to be expected there. Try passing an explicit splitter (even set a random_state) so you're sure to be comparing the same things. Also, max_iter=10 seems awfully low; do you get convergence warnings? $\endgroup$ – Ben Reiniger Sep 21 '20 at 15:41
  • $\begingroup$ the optimal parameter according to GridSearchCV was max iter 10... also the scores are not the best score, but accuracy score calculated by me. What do you mean for explicit splitter? Do you think the parameters I set are right? $\endgroup$ – Anna Sep 21 '20 at 16:09
0
+50
$\begingroup$

You are implementing the same model but with different optimization techniques, which means the final weights will be different but if the problem is convex it should converge to the same value.

  • SGD is dependent on initialization (as most iterative processes);
  • The convergence might take way too long is learning rates are not set properly;
  • Since SGD uses batches instead of optimizing on the entire dataset at once, it might not reach global optimum for the whole training dataset.

SGD might give better final scores since:

  • Your final scoring metric optimum might differ from loss function optimum;
  • Test/Validation optimum will probably differ from training optimum.

Also, as pointed by N. Kiefer it is worth to check if the 0.4 points difference on scores are really statistically significant.

$\endgroup$
1
$\begingroup$

When used with loss="hinge" The SGDClassifier gives a LinearSVM, so they should be the same. This is matter of choosing the same hyperparameters for both. Can you check that you using the exact same parameters?

As a side note (I don't know your dataset), 41.1 and 41.5 looks pretty similar, this also might be about splitting the training/testing data exactly the same and stopping the training at the correct time.

$\endgroup$
1
  • $\begingroup$ The hyperparameters are there as you can see, commented $\endgroup$ – Anna Sep 21 '20 at 13:45
1
$\begingroup$

Remember that

LinearSVM uses the full data and solve a convex optimization problem with respect to these data points.

SGDClassifier can treat the data in batches and performs a gradient descent aiming to minimize expected loss with respect to the sample distribution, assuming that the examples are iid samples of that distribution.

As a working example check the following and consider:

  1. Increasing the number of iterations
  2. Increase the number of iterations before the early stoping
  3. Both classifier should be using the same loss function, in this case "squared hinge"
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y= True)
X_train ,X_test, y_train, y_test = train_test_split(X, y, test_size = .3, random_state = 42)

sgd = SGDClassifier(random_state= 42,loss = "squared_hinge", max_iter= 100000,n_iter_no_change=1000).fit(X_train, y_train)
linearsvm = LinearSVC(random_state= 42,max_iter= 100000).fit(X_train, y_train)

sgd.score(X_test, y_test)
linearsvm.score(X_test,y_test)

Before modifying these two parameters, SGDClassifier gave ~ 20% less accuracy After both performed the same on the test set

$\endgroup$
5
  • $\begingroup$ Thanks, but why the squared hinge in the SDGClassifier? Also, C parameters etc. need to be fine tuned, default ones are of no use $\endgroup$ – Anna Sep 22 '20 at 8:44
  • $\begingroup$ I also tried increasing the number of iterations, but to no avail $\endgroup$ – Anna Sep 22 '20 at 12:45
  • $\begingroup$ Ok, so I see there is no much more I can do for helping you, but If you want to undestand why the results are so different the answers lies here..."LinearSVM uses the full data and solve a convex optimization problem with respect to these data points. SGDClassifier can treat the data in batches and performs a gradient descent aiming to minimize expected loss with respect to the sample distribution, assuming that the examples are iid samples of that distribution" $\endgroup$ – Julio Jesus Sep 22 '20 at 20:56
  • $\begingroup$ The "squared_hinge" is the default on the LinearSVM that's why I'm using it on SGDClassifier $\endgroup$ – Julio Jesus Sep 22 '20 at 20:57
  • $\begingroup$ I understand, thank you, I didn't realise it! $\endgroup$ – Anna Sep 22 '20 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.