My data has an extreme class imbalance - 99.7% is 0's and 0.2% is 1's and almost all the predictor variables (6 out of 7) are categorical. I trained an xgboost classifier after performing an upsampling (using ROSE in R). Evaluated the model on 20% test data and the following is the result,
Recall, as you can see, is not great. Additionally, if I run the model on a completely different test data (meaning not a chunk of the original data), then the recall value drops to exactly 50%. Is this happening because it is overfitting? Can someone please let me know how I can better this model, considering the extreme class imbalance?
I'm a newbie in the field. Any help would be much appreciated. Thanks!