1
$\begingroup$

I am testing different models (SVM, Logistic Regression, Naive Bayes, Random Forest) for predicting the class of a spam email. My target is a binary variable. I am analysing only text, no other fields. My dataset includes

Label  
0.0    3333
1.0     768

As you can see there is a big problem with classes imbalanced. I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset. I got good results in terms of F1, recall and accuracy for upsampling (above 88%; max 97%), bad for downsampling (<=76%). For instance:

Down
              precision    recall  f1-score   support

         0.0       0.79      0.43      0.56       102
         1.0       0.61      0.87      0.76       114


Confusion Matrix: 
 [[ 49  60]
 [ 12 100]]


Up
              precision    recall  f1-score   support

         0.0       1.00      0.85      0.91       873
         1.0       0.87      1.00      0.94       884



Confusion Matrix: 
 [[772 141]
 [  20 822]]

I would like to ask you if these values can be considered good results or they can't. I am considering a publication (not only to include similar analysis), so I would like to check if such results can be considered reliable, despite of the imbalance.

Any suggestions and advice will be greatly welcome.

$\endgroup$
1
2
+50
$\begingroup$

There seems to be a mistake in your method:

I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset.

It's incorrect to change the distribution of the test set. When resampling, the resampling should be applied only on the training set. The goal is to force the model to take into account the two classes, because in case of imbalance the model tends to focus on the majority class. However the true proportion of the class in the "real dataset" is still the same, and the test set should follow this true proportion. Otherwise the performance looks artificially good on the test set, even though the classifier will make more mistakes with real data since it doesn't have the same distribution.

So the performance values that you obtain on a resampled dataset are meaningless, I'm afraid.

I am considering a publication (not only to include similar analysis), so I would like to check if such results can be considered reliable, despite of the imbalance.

If you are considering a peer-reviewed publication, you must also make sure that your contribution is original (i.e. new) and brings some advantage over the existing methods. This means that you need to know the state of the art in spam classification (there are a lot of papers already published about this task) and show what your method improves something compared to the existing methods. Ideally this is done by proving that your new method obtains better performance than the state of the arts methods using a benchmark dataset. But it's usually hard to beat state of the art performance on a well-known problem.

$\endgroup$
11
  • 1
    $\begingroup$ But did your two subsets come from the same source? The standard process would be to collect a large number of emails from diverse sources over a given period of time, then label them as either spam or not. This way the proportion of spam in the dataset is roughly the same as what what it would be in real conditions, when the spam filter is applied to all the incoming email. If the two subsets don't come from the same source, for example if they were collected separately, then the dataset is biased from the start. Of course you can play with it for testing/educational purposes, but it ... $\endgroup$ – Erwan Nov 16 '20 at 23:26
  • 1
    $\begingroup$ wouldn't be considered a reliable dataset. Is this work part of your PhD? Is your topic about NLP or similar? It's normal to spend some time in the first year discovering problems and methods so it's fine, but if this your main PhD work I hope that you're aware that it's a field which progresses very fast (I can't keep track myself ;) ), nowadays it's very likely that the state of the art in spam filtering requires more advanced methods like Deep Learning. I don't know the context of your work but in general there's little chance to outperform the state of the art with standard methods on ... $\endgroup$ – Erwan Nov 16 '20 at 23:36
  • 1
    $\begingroup$ ... this kind of problem. Obviously I don't know anything about your work so take this with a grain of salt, but I'd suggest you make sure (typically with your supervisor) that your PhD direction/topic is original enough to lead to publishing material. For example ideas such as resampling, feature engineering with punctuation/case are very standard techniques which have probably been tried before, so I doubt it's going to be enough to publish in a decent conference/journal. $\endgroup$ – Erwan Nov 16 '20 at 23:40
  • 1
    $\begingroup$ Note that you could probably find other datasets for the task for example from the TREC shared tasks. There might be other more recent available datasets as well, these would certainly be mentioned in the publications about the task. $\endgroup$ – Erwan Nov 16 '20 at 23:54
  • 1
    $\begingroup$ ... supervision, it's important to address the issue as soon as possible because you don't want to spend too much PhD time working in a "wrong" direction ("wrong" = not sure to lead to publications and PhD degree). Sometimes a department head can help finding a co-supervisor who is expert in the domain. Be aware that I'm just giving you random advice here as I don't know your situation, also this site is not meant for this kind of advice anyway :) Btw there's AcademiaSE for questions about academia. So use your best judgement, and good luck for the PhD :) $\endgroup$ – Erwan Nov 21 '20 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.