# Metrics show badly performing model for multiclass

So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and bigram counts as features, plus string length and entropy.

I first remove any 0 variance features (~1M -> 2k feats.), then using sklearn.decomposition.TruncatedSVD I reduce it further to 500 feats. The data is then split to 10k samples batches (min 500 samples/class), following the distribution of each class in the raw data. Splits continue this way until all samples from the largest class has been added to a split.

Then I run imblearn.under_sampling.RandomUnderSampler (100k samples for majority classes) and imblearn.over_sampling.SMOTE (50k) for minority classes) for under- and oversampling on each of the files to fix the imbalance.

After this I train several models and then compare different metrics to get a better idea of which is the best choice:

• sklearn.linear_model.SGDClassifier(loss='log', alpha=1e-4,tol=1e-4,penalty='elasticnet',max_iter=1000,shuffle=True)
• sklearn.linear_model.Perceptron(alpha=1e-3, penalty=None,tol=1e-4,max_iter=1000,shuffle=True)
• sklearn.linear_model.PassiveAggressiveClassifier(C=1.0,tol=1e-4,max_iter=1000,shuffle=True)
• sklearn.neural_network.MLPClassifier(solver='adam',learning_rate='adaptive',activation='tanh',validation_fraction=0.05,alpha=1e-5,learning_rate_init=1e-4,max_iter=20,hidden_layer_sizes=(100,))
• sklearn.naive_bayes.BernoulliNB(alpha=1.0e-5, fit_prior=True)

As you can see in the picture below the metrics show that the classifier is not working optimally. As far as I have understood from searching around, AUC and cohen_kappa are the more realiable metrics for multiclass and imbalanced data sets.

I am concerned that the AUC metric show that all models are doing well, but the others, in particular cohen_kappa which is significantly lower (I know though that it is hard to directly compare the metrics).

Am I doing any obvious mistakes in this, or any obvious ways I can improve performance/metric scores?