I have a dataset containing thousands of text posts. I am building a binary classifier that will classify posts as safe (0) or risky (1). I randomly picked some of them and manually labeled. Label 1 is minority.
Imbalanced data results in skewed performance. To overcome this, I undersample to obtain a 50%/50% distribution for both train and test set.
- If I use stratified strategy (in scikit-learn) for ZeroR, it acts like random guesser. Is this a good baseline?
If I use "guess 1" all the time, then its recall is 100% all the time. However recall is the most important metric for me. So other algorithms' predictions look unsuccessful. Should I ignore recall when using ZeroR and only compare accuracy?
I also would like to apply 10-fold cross validation, should I apply undersampling before splitting as well?
- Or instead should I go with F1-Score without undersampling? This time ZeroR has high F1 score due to 100% Recall.