I am currently working with a dataset that is imbalanced, about 30k rows * 14 features (just for you know), and 99.5% of the data is labeled 0. Since the model is strongly imbalanced I decided to use precision/recall/f1 score to decide the performance of model.
I used SMOTE to oversample my training data (after splitting the validation set out). Now my model is trained with oversampled training data, and I am going to test it with validation set. If I just validate it on original validation data, I get a F1 score around 0.05, and the classification report is followed:
precision recall f1-score support Class 0 1.00 0.86 0.93 7606 Class 1 0.03 0.75 0.05 36
If I oversample my validation data, I get a F1 score around 0.85:
precision recall f1-score support Class 0 0.84 0.86 0.85 7606 Class 1 0.86 0.83 0.85 7606
My question is:
Should I use an oversampled validation set? (because the result is much prettier but I think the model is the same anyway)
Why do I have such bad metrics on the original validation data? Is it because the data size is not big enough?