0
$\begingroup$

I have a dataset containing thousands of text posts. I am building a binary classifier that will classify posts as safe (0) or risky (1). I randomly picked some of them and manually labeled. Label 1 is minority.

Imbalanced data results in skewed performance. To overcome this, I undersample to obtain a 50%/50% distribution for both train and test set.

  • If I use stratified strategy (in scikit-learn) for ZeroR, it acts like random guesser. Is this a good baseline?
  • If I use "guess 1" all the time, then its recall is 100% all the time. However recall is the most important metric for me. So other algorithms' predictions look unsuccessful. Should I ignore recall when using ZeroR and only compare accuracy?

  • I also would like to apply 10-fold cross validation, should I apply undersampling before splitting as well?

  • Or instead should I go with F1-Score without undersampling? This time ZeroR has high F1 score due to 100% Recall.
$\endgroup$
  • $\begingroup$ Is there any specific reason for you to not to try over sampling? $\endgroup$ – Toros91 Nov 16 '17 at 1:18
  • $\begingroup$ I don't know if I can oversample text posts. I am also worried of the overfitting risk. Other than that, I don't have any other reason. $\endgroup$ – aladagemre Nov 16 '17 at 9:16
  • 1
    $\begingroup$ if you undersample you will might loose some information. I would suggest you to give a try with oversampling for 1 set of data and see how it works. There are many package like SMOTE, ROSE and many more. Overfitting happens if you just duplicate the same record, but SMOTE and ROSE don't do that. $\endgroup$ – Toros91 Nov 16 '17 at 9:19
1
$\begingroup$

You should only undersample the data that is used for training. Test data should represent the true distribution.

Then you have at least two possible baselines. One is ZeroR (your second point), which always predicts the majority class. Given a true distribution of (0) 10% and (1) 90%, this would give you a recall of 100% for (1) and 0% for (0) and 90% accuracy.

If rows represent the true labels and the columns the predictions, the confusion matrix looks something like this:

$\begin{bmatrix}90 & 0\\10 & 0\end{bmatrix}$

The other would be to flip a coin with probability 90% for class (1) (your first point but without undersampling), which gives you a recall of 90% for (1) and 10% for (0), and 82% accuracy, which is a little more balanced:

$\begin{bmatrix}81 & 9\\9 & 1\end{bmatrix}$

If your classes are strongly imbalanced then recall and accuracy have the same problems as you noted above. F1 might be a better choice, but the best metric depends on your application.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you so much! I forgot to say, the data distribution is not natural. I've arranged them. Indeed in real world, they may have 0.001% occurrence frequency (I don't know) but in my case they are ~20% because I hand picked them from related forum sections due to their usefulness (relatedness) in my problem. It is no good to use all the dataset (forum sections) since there are too many irrelevant stuff. So in this case, there's no way to have a natural imbalanced data. Should this lead me to using a balanced dataset for testing? $\endgroup$ – aladagemre Nov 16 '17 at 9:15
  • $\begingroup$ I think that doesn't necessarily change the approach it just makes it more difficult.. you'd still want your test set to be representative of what you will encounter when you apply the model to the "real world". you probably want to collect enough data to have a reasonable number of positive cases in your test set... $\endgroup$ – oW_ Nov 17 '17 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.