1
$\begingroup$

I am struggling with confusion matrices and their outputs. I thought to follow all the steps right, but unfortunately it seems that something is not going well.

I had a dataset built and labelled on my own. It shows a class imbalance so I decided to apply undersampling and oversampling, looking at F1-score and Recall as in many papers and on the web. The steps were:

  • split data in train and test (80/20)
  • apply resampling only on train set
  • apply pre-processing algorithm (BoW, TF-IDF, ...)
  • use different classifiers to get results
  • look at performance using confusion matrices (or alternatively ROC)

I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection). In theory, if I extract features only from text, I get a better score rather than extracting also features from email addresses. I would like to ask you for some tips and suggestions, if you have any, as I think that this cannot be possible, as the expected results should be higher in the second case, when I consider also information from email address (number of dots, suffix, registration date,...).

I am thinking at a problem of overfitting or some other issues with model building. I would appreciated if you could tell me your thoughts on this.

Thank you for all your help.

$\endgroup$

1 Answer 1

1
$\begingroup$

I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection).

Typically this would happen if the model is overfit: not enough data and/or too many features make the model pick patterns which happen by chance in the training data.

Usually with text one has to remove the least frequent words in order to avoid overfitting. You might also want to check the additional features, remove anything which happens too rarely.

Also, the confusion matrix, has given me weird outputs

       0      1
0    [[2036  161]
1    [   1 2196]]

Observations:

  • True class 0 has 2036+161 = 2197 instances, true class 1 has 1+2196=2197 instances. These results are obtained with the resampled data.
  • Assuming class 1 is positive: 2196 True Positives (TP), 2036 TN, 161 FP (true positive predicted as negative) and 1 FN (true negative predicted as positive).
    • recall = 0.999, precision = 0.932. That's an f1-score somewhere higher than 0.95 (probably due to the resampled data).
  • The second confusion matrix is also clearly obtained with the resampled data, and it shows perfect performance (F1-score is 1).

These matrices show the performance obtained on the resampled data, so it's similar to the performance on the training data. Since the performance on a real test set is much lower, this confirms strong overfitting.

$\endgroup$
5
  • $\begingroup$ Thanks Erwan. So do you suggest to reduce the number of features in order to get similar performance? I only applied tf-idf removing stopwords to be honest and some punctuation. What would you suggest to use (eg. word frequency, removing words having less than 3 characters, containing numbers,...)? $\endgroup$
    – V_sqrt
    Feb 21, 2021 at 15:18
  • 1
    $\begingroup$ @Val start by removing the words which appear in less than N documents (i.e. threshold on minimum doc frequency), varying N to find the optimal threshold. In case you're interested I gave a detailed explanation about this here. You might also want to start with an experiment without resampling: first because resampling adds a layer of complexity so it's harder to understand what's going on, and second because it doesn't always work better with resampling. $\endgroup$
    – Erwan
    Feb 21, 2021 at 15:41
  • $\begingroup$ Thanks a million, Erwan. I observed a few words that I can remove (for example international words in Chinese, Arabic, and so on; only numbers; some stopwords that was not captured from the package,...). I thought that in case of spam/not spam detection, some rare words might be useful for determining which one was used in either one or other dataset, but I think I was wrong. Another point: what do you think of using TextFeatureSelection in Python? it gives X2 values so maybe I can just looking at the highest values to select features (after applied your filters) $\endgroup$
    – V_sqrt
    Feb 21, 2021 at 15:51
  • $\begingroup$ @Val it can happen that some rare words help indeed, but using rare words is risky because for any statistical process to be reliable one needs a representative sample, and by definition the sample is small in the cases of rare words. Since there are many rare words, it's unavoidable that a lot of them happen just by chance, and whenever the model uses a pattern which happens by chance in the training set it overfits. $\endgroup$
    – Erwan
    Feb 21, 2021 at 17:36
  • 1
    $\begingroup$ Feature selection is relevant yes, I don't know this particular library but it looks suitable for your case. Be careful that the "first method" is based on measuring the "correlation" between each individual feature and the target: in the case of rare words it would often happen that there is strong correlation by chance, so this method would not help with this problem in general. However the second GA method works by trying different subsets of features, so this one should be able to discard "bad" features, but it's more advanced and it requires more computation. $\endgroup$
    – Erwan
    Feb 21, 2021 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.