I am struggling with confusion matrices and their outputs. I thought to follow all the steps right, but unfortunately it seems that something is not going well.
I had a dataset built and labelled on my own. It shows a class imbalance so I decided to apply undersampling and oversampling, looking at F1-score and Recall as in many papers and on the web. The steps were:
- split data in train and test (80/20)
- apply resampling only on train set
- apply pre-processing algorithm (BoW, TF-IDF, ...)
- use different classifiers to get results
- look at performance using confusion matrices (or alternatively ROC)
I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection). In theory, if I extract features only from text, I get a better score rather than extracting also features from email addresses. I would like to ask you for some tips and suggestions, if you have any, as I think that this cannot be possible, as the expected results should be higher in the second case, when I consider also information from email address (number of dots, suffix, registration date,...).
I am thinking at a problem of overfitting or some other issues with model building. I would appreciated if you could tell me your thoughts on this.
Thank you for all your help.