1
$\begingroup$

I am trying to build a model that predicts if an email is spam/not-spam. After building a logistic regression model, I have got the following results:

          precision    recall  f1-score   support

         0.0       0.92      0.99      0.95       585
         1.0       0.76      0.35      0.48        74

    accuracy                           0.92       659
   macro avg       0.84      0.67      0.72       659
weighted avg       0.91      0.92      0.90       659

Confusion Matrix: 
 [[577   8]
 [ 48  26]]

Accuracy:  0.9150227617602428

The F1-score is the metric I am looking at. I am having difficulties in explaining the results: I think are very bad results! May I ask you how I could improve it? I am currently considering a model that looks at corpus of the emails (subject + corpus).

After Erwan's answer:

I oversampled the dataset and these are my results:

Logistic regression
              precision    recall  f1-score   support

         0.0       0.94      0.77      0.85       573
         1.0       0.81      0.96      0.88       598

    accuracy                           0.86      1171
   macro avg       0.88      0.86      0.86      1171
weighted avg       0.88      0.86      0.86      1171

Random Forest
              precision    recall  f1-score   support

         0.0       0.97      0.54      0.69       573
         1.0       0.69      0.98      0.81       598

    accuracy                           0.77      1171
   macro avg       0.83      0.76      0.75      1171
weighted avg       0.83      0.77      0.75      1171
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In your results you can observe the usual problem with imbalanced data: the classifier favors the majority class 0 (I assume this is class "ham"). In other words it tends to assign "ham" to instances which are actually "spam" (false negative errors). You can think of it like this: with the "easy" instances, the classifier gives the correct class, but for the instances which are difficult (the classifier "doesn't know") it chooses the majority class because it's the most likely.

There are many things you could do:

  • Undersampling the majority class or oversampling the minority class is the easy way to deal with class imbalance.
  • Better feature engineering is more work but it's often how to get the best improvement. For example I guess that you use all the words in the emails as features right? So you probably have too many features and that probably causes overfitting, try reducing dimensionality by removing rare words.
  • Try different models, for instance Naive Bayes or Decision Trees. Btw Decision Trees are a good way to investigate what happens inside the model.
Improve this answer
$\endgroup$
8
  • $\begingroup$ Hello Erwan, thank you so much for your answer and tips. I have tried to apply the first and the last bullets in your answer and I think the results are slightly improved. Could you please confirm and let me know what you think, if you think the better feature engineering would improve it further or if you think it overfits? Thanks a lot $\endgroup$
    – LdM
    Dec 7, 2020 at 0:45
  • 1
    $\begingroup$ @LdM: resampling is simple but there's something to be careful about: here I assume you have split between training/test set after resampling, right? That's not correct because this way the test set doesn't follow the true proportion, so there's a good chance that the performance is over-estimated. You should split first, resample on the training set only, then evaluate on the test set. $\endgroup$
    – Erwan
    Dec 7, 2020 at 0:59
  • $\begingroup$ thanks, Erwan. So what I did was: smote_over_sample = SMOTE(sampling_strategy='minority') # Testing Count Vectorizer X, y = bag_of_words(df) X, y = smote_over_sample.fit_resample(X, y) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=40) smote_result = smote_result.append(training_log(X_train, X_test, y_train, y_test, 'Count Vectorize'), ignore_index = True) Is it wrong? I have followed this example: kaggle.com/ruzarx/oversampling-smote-and-adasyn $\endgroup$
    – LdM
    Dec 7, 2020 at 1:02
  • 1
    $\begingroup$ Yes, this is wrong, the test results are biased this way. it's a common error, we see it regularly on DataScienceSE, for instance in this question. See also this one. $\endgroup$
    – Erwan
    Dec 7, 2020 at 1:16
  • 1
    $\begingroup$ @LdM Sorry I don't really understand the issue? maybe because I'm not very familiar with scikit functions. I can only tell you the principle: (1) split training/test; (2) apply resampling on the training set, so that the model is trained with balanced data (note that you may do some evaluation on the training set or some part of it if you want, but that's not the final evaluation); (3) apply model to test set and evaluate on it. the test set is still imbalanced, as it's supposed to be. normally you should see less false negative errors but probably more false positive errors. $\endgroup$
    – Erwan
    Dec 7, 2020 at 1:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.