Metrics show badly performing model for multiclass

Question

So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and bigram counts as features, plus string length and entropy.

I first remove any 0 variance features (~1M -> 2k feats.), then using sklearn.decomposition.TruncatedSVD I reduce it further to 500 feats. The data is then split to 10k samples batches (min 500 samples/class), following the distribution of each class in the raw data. Splits continue this way until all samples from the largest class has been added to a split.

Then I run imblearn.under_sampling.RandomUnderSampler (100k samples for majority classes) and imblearn.over_sampling.SMOTE (50k) for minority classes) for under- and oversampling on each of the files to fix the imbalance.

After this I train several models and then compare different metrics to get a better idea of which is the best choice:

• sklearn.linear_model.SGDClassifier(loss='log', alpha=1e-4,tol=1e-4,penalty='elasticnet',max_iter=1000,shuffle=True)
• sklearn.linear_model.Perceptron(alpha=1e-3, penalty=None,tol=1e-4,max_iter=1000,shuffle=True)
• sklearn.linear_model.PassiveAggressiveClassifier(C=1.0,tol=1e-4,max_iter=1000,shuffle=True)
• sklearn.neural_network.MLPClassifier(solver='adam',learning_rate='adaptive',activation='tanh',validation_fraction=0.05,alpha=1e-5,learning_rate_init=1e-4,max_iter=20,hidden_layer_sizes=(100,))
• sklearn.naive_bayes.BernoulliNB(alpha=1.0e-5, fit_prior=True)

As you can see in the picture below the metrics show that the classifier is not working optimally. As far as I have understood from searching around, AUC and cohen_kappa are the more realiable metrics for multiclass and imbalanced data sets.

I am concerned that the AUC metric show that all models are doing well, but the others, in particular cohen_kappa which is significantly lower (I know though that it is hard to directly compare the metrics).

Am I doing any obvious mistakes in this, or any obvious ways I can improve performance/metric scores?

Answer 1

I think you need to consider these metrics also i.e., Precision and Recall when you decide on some model.

As you can in the above outcome that there are Precision and Recall values, you din't explain anything about the problem's business perspective.

For example: Consider a data set with 100,000 observations. A data set consist of candidates who applied for Internship in Harvard. Apparently, harvard is well-known for its extremely low acceptance rate. The dependent variable represents if a candidate has been shortlisted (1) or not shortlisted (0). It was found ~ 98% did not get shortlisted and only ~ 2% got lucky. In this scenario you need to understand that precision and recall plays a vital role at the time of selecting model.

In the same way if your business problem is inline with the above scenario then you need to select a model with high Precision and Recall.

In the above screenshot which you have shared, it think you have good performance scores too(Assuming that your model is not over fitting). So, you can select which ever model suits your business problem.

Hope my answer is helpful!