How to evaluate model with imbalanced data binary classification?

Question

I have a binary classification problem. I am using Area under precision recall curve as the evaluation metric. The dimensions of my data are (211, 1361). The data is imbalanced so I have used various sampling techniques to address the imbalance problem. I divided the data into train and test sets.

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state=RANDOM_STATE, stratify = y)

train set (147, 1361)
test set (64, 1361) 

I am using the following classifiers for this task:

RegressionModel    = LogisticRegression(random_state=RANDOM_STATE, penalty='l1')
RandomForrestModel = RandomForestClassifier(random_state=RANDOM_STATE)
ExtraTreesModel    = ExtraTreesClassifier(random_state=RANDOM_STATE)
AdaBoostModel      = AdaBoostClassifier(random_state=RANDOM_STATE)
SVM = SVC(random_state=RANDOM_STATE, probability=True)
mlp = MLPClassifier(random_state=RANDOM_STATE)
GradientBoostModel = GradientBoostingClassifier(random_state=RANDOM_STATE)

I then evaluated all the models using training data with different sampling strategies(Random oversampling, Random under sampling, SMOTE, ADASYN) using stratifies k fold cross validation with k = 10. I computed mean score and standard deviation on test score to see how varied my predictions are across 10 folds. The resampling is done inside cross validation.

     RO_train    RO_test    RO_std     SMOTE_train  SMOTE_test  SMOTE_std   ADASYN_train ADASYN_test ADASYN_std RU_train    RU_test     RU_std

ADB 1.000000    0.922866    0.173459    1.000000    0.875644    0.226287    1.000000    0.899255    0.177378    1.000000    0.858810    0.168208
ET  1.000000    0.952778    0.116567    1.000000    0.940903    0.092280    1.000000    0.907361    0.159184    1.000000    0.904167    0.129441
GB  1.000000    0.898710    0.224462    1.000000    0.888294    0.139198    1.000000    0.921210    0.118537    1.000000    0.884722    0.157905
LR  1.000000    0.670480    0.322062    1.000000    0.653813    0.312906    1.000000    0.642494    0.303857    1.000000    0.556829    0.246005
RF  1.000000    0.858611    0.211937    1.000000    0.877778    0.148527    1.000000    0.939167    0.106592    1.000000    0.818889    0.159638
SVM 1.000000    0.652461    0.088840    1.000000    0.653519    0.088681    1.000000    0.653519    0.088681    0.266960    0.580982    0.011825
mlp 0.892356    0.676389    0.219603    0.873744    0.624167    0.227717    0.891931    0.702262    0.192916    0.787599    0.498948    0.240001

I then calculated some statistics based on above cv results.

max std: LR 0.32206202775943316 Random Oversampling_test_std
min std: SVM 0.08884005857118661 Random Oversampling_test_std
max std: LR 0.3129063718366338 SMOTE_test_std
min std: SVM 0.0886812284028803 SMOTE_test_std
max std: LR 0.30385683057394436 ADASYN_test_std
min std: SVM 0.0886812284028803 ADASYN_test_std
max std: LR 0.2460049718614556 Random undersampling_test_std
min std: SVM 0.011825217886416502 Random undersampling_test_std

After doing cv, I evaluated my models on test data set which was not part of cv. I have not done any kind of resampling on this data.

Random Oversampling SMOTE   ADASYN  Random undersampling
ADB 0.810964    0.900719    0.974892    0.855232
ET  0.987879    0.898830    0.927410    0.988636
GB  0.809401    0.899905    0.902631    0.801022
LR  0.505468    0.585471    0.521210    0.679356
RF  0.971350    0.955276    0.979021    0.913223
SVM 0.628788    0.630200    0.630200    0.585938
mlp 0.773493    0.748242    0.670285    0.711538

here is the confusion matrix on test data:

    Random Oversampling       SMOTE              ADASYN        Random undersampling
ADB [[11, 0], [5, 48]]  [[11, 0], [5, 48]]  [[10, 1], [3, 50]]  [[10, 1], [7, 46]]
ET  [[9, 2], [0, 53]]   [[9, 2], [2, 51]]   [[9, 2], [1, 52]]   [[11, 0], [3, 50]]
GB  [[8, 3], [3, 50]]   [[8, 3], [2, 51]]   [[7, 4], [2, 51]]   [[11, 0], [6, 47]]
LR  [[10, 1], [7, 46]]  [[10, 1], [8, 45]]  [[10, 1], [7, 46]]  [[10, 1], [9, 44]]
RF  [[10, 1], [1, 52]]  [[9, 2], [1, 52]]   [[11, 0], [2, 51]]  [[11, 0], [3, 50]]
SVM [[0, 11], [0, 53]]  [[0, 11], [0, 53]]  [[0, 11], [0, 53]]  [[0, 11], [0, 53]]
mlp [[11, 0], [8, 45]]  [[8, 3], [4, 49]]   [[9, 2], [5, 48]]   [[11, 0], [15, 38]]

Non-ensemble classifiers didn't perform well so I decided to go with ensemble method.

I decided to go with [RF + ADASYN] as it gave 100% train score, 93% cross validated AUPRC and 97% on final unseen data.

When I trained RF on full ADASYN data and tested it on unseen test set. It performed quite poorly. Since I have quite few data, I am suspecting that may be it is underfitting but it's just a hunch. Any suggestions in this regard would be helpful.

Answer 1

Things are fine in your approach.

I would suggest an EDA on the data if you have not done to see e.g. how diffrent classes are distributed in 2D using differen methods. You have Low-Sample-Size Problem i.e. 1150 dimensions in your data have 0 information. Regardless of how different algorithms cope with high dimensional data, I suggest to remove that big fraction of information-less dimensions.

I do not know what you mean by "resampling is done inside CV". If you explain it more maybe we can discuss, but in general if you produce your resampled data first and simply do CV, you will be on the safe side. Some resamplings (like SMOTE) work on the topology of data points (simply putting a point between some others in n-dimensional space) and in your case, 211 points in 10-k CV means 20 per fold. It can not represent the geometry of data well probably. I would suggest doing resampling first keeps you on safe side.

The last but not least is LOO (Leave-One-Out, simply 211-fold CV in your case). In such a case like yours, I would do that as well.

At the end, I am interested to see a 2D projection of your data. Maybe then we can talk about it more insightful.