1
$\begingroup$

I'm trying to fit a GaussianNB and a LinearSVC to binary labeled text using scikit-learn. To do that, I'm using a TfidfVectorizer to transform my sentences into a matrix of features.

This is essentially my code:

train, test = train_test_split(data, test_size=0.3)
vectorizer = TfidfVectorizer()
vectorizer.fit(data["Document"])
tfidf_wm_train = vectorizer.transform(train["Document"])

gauss = GaussianNB()
gauss.fit(tfidf_wm_train.toarray(), train["Class"])


tfidf_wm_test = vectorizer.transform(test["Document"])

train_guess = gauss.predict(tfidf_wm_train.toarray())
guess = gauss.predict(tfidf_wm_test.toarray())
print("GaussianNB:")
print(classification_report(train["Class"], train_guess))
print(classification_report(test["Class"], guess))

The output:

GaussianNB:
              precision    recall  f1-score   support

       rural       0.68      0.60      0.64       321
     science       0.63      0.71      0.67       309

    accuracy                           0.65       630
   macro avg       0.66      0.65      0.65       630
weighted avg       0.66      0.65      0.65       630

              precision    recall  f1-score   support

       rural       0.11      0.11      0.11       129
     science       0.18      0.18      0.18       141

    accuracy                           0.14       270
   macro avg       0.14      0.14      0.14       270
weighted avg       0.14      0.14      0.14       270

And I just don't understand how the classifier can only get 14% accuracy on the test set. This of course means that the classifier is able to give the wrong result in 86% of cases. And given that this is a binary classification problem, I can get a 86% accuracy by just flipping the result labels. Which is a higher accuracy than the classifier reaches on the training set.

And the exact same happens for LinearSVC:

LinearSVC:
Train:
               precision    recall  f1-score   support

       rural       0.65      0.64      0.65       313
     science       0.65      0.66      0.66       317

    accuracy                           0.65       630
   macro avg       0.65      0.65      0.65       630
weighted avg       0.65      0.65      0.65       630

Test:
               precision    recall  f1-score   support

       rural       0.16      0.15      0.15       137
     science       0.14      0.14      0.14       133

    accuracy                           0.15       270
   macro avg       0.15      0.15      0.15       270
weighted avg       0.15      0.15      0.15       270

Can somebody explain to me what's going on?

This is not something that happened by bad luck while splitting test and train data. This happens consistently ..

$\endgroup$

1 Answer 1

1
$\begingroup$

Gaussian Naive Bayes

GaussianNB assumes that your features (input data) follow a normal distribution (or Gaussian distribution) for both classes of your binary label.

However, looking at your code, it looks like you are only using TfidfVectorizer to prepare your data. You need to further process it to get it to follow a normal distribution. This step cannot be skipped as you must respect GaussianNB's assumption about the data, otherwise you will get the sub-optimal results you are observing.

To confirm that this is the reason you are getting poor accuracy, you can manually check the distribution of your features (tf-idf values) for both classes

Alternative: Linear Support Vector Classification

You mentioned using LinearSVC in your question, but I don't see the code or performance for it. I think it will be a more optimal solution, as GaussianNB is not the best choice for high-dimensional classification tasks like yours (td-idf matrices lead to a lot of dimensions). Plus you wouldn't need to transform your data to follow a normal distribution as the LinearSVCs do not make that same assumption like GaussianNB.

Additionally, LinearSVC would not be sensitive to irrelevant features like GaussianNB is, it works better with sparse data (many 0s) which you are likely to have in your td-idf matrix and it handles class imbalance better.

In short, LinearSVC naturally performs better on this kind of tasks compared to GaussianNB and will allow you to drop the requirement to transform your data, which should lead to better performance.

$\endgroup$
2
  • $\begingroup$ Thank you for this detailed answer! I indeed forgot to account for normal distribution.. And I added my results for LinearSVC, but it shows the same trend.. There are definitely several reasons why my approach isn't optimal. But what is really confusing me is how the classification with switched labels works better on the test set than the original does on the training set. $\endgroup$
    – Towdo
    Commented Nov 21, 2023 at 12:30
  • 1
    $\begingroup$ @Towdo what's the balance of your binary target? I suspect it's probably 14/86% and your model is predicting 100% the same class, which will give you either 14% accuracy or 86%. $\endgroup$ Commented Nov 21, 2023 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.