So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and bigram counts as features, plus string length and entropy.
I first remove any 0 variance features (~1M -> 2k feats.), then using
sklearn.decomposition.TruncatedSVD I reduce it further to 500 feats. The data is then split to 10k samples batches (min 500 samples/class), following the distribution of each class in the raw data. Splits continue this way until all samples from the largest class has been added to a split.
Then I run
imblearn.under_sampling.RandomUnderSampler (100k samples for majority classes) and
imblearn.over_sampling.SMOTE (50k) for minority classes) for under- and oversampling on each of the files to fix the imbalance.
After this I train several models and then compare different metrics to get a better idea of which is the best choice:
As you can see in the picture below the metrics show that the classifier is not working optimally. As far as I have understood from searching around,
cohen_kappa are the more realiable metrics for multiclass and imbalanced data sets.
I am concerned that the AUC metric show that all models are doing well, but the others, in particular
cohen_kappa which is significantly lower (I know though that it is hard to directly compare the metrics).
Am I doing any obvious mistakes in this, or any obvious ways I can improve performance/metric scores?