0
$\begingroup$

I am training a classification model with 3 classes using a deep neural network.

The classes have been resampled and balanced.

I have around 600000 samples... equally distributed.

The dataset is also divided equitably in the train/test/validation dataset.

After training, the overall accuracy is ~65% but individual classes have a disparity.

Class 0 and 1 have high precision and recall, but the class 2 has very low precision and recall... How can I fix this...

Model defined:


model = tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=(20, 4)),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(1024, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(1024, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(3, activation='softmax'),
])

hist = evaluate(
    model=model,
    train_data=(X_train, Y_train),
    val_data=(X_test, Y_test),
    epochs=100,
    batch_size=256,
    verbose=True,
    loss='categorical_crossentropy',
    metrics=['accuracy'],
)

LR scheduler, Early Stopping has been implemented.

Classification report:

              precision    recall  f1-score   support

           0       0.66      0.99      0.79    196323
           1       0.68      0.99      0.80    196323
           2       0.57      0.03      0.06    196323

    accuracy                           0.67    588969
   macro avg       0.64      0.67      0.55    588969
weighted avg       0.64      0.67      0.55    588969

Confusion Matrix

[[193909    177   2237]
 [   298 193918   2107]
 [ 98184  92287   5852]]
  • accuracy_score: 0.6684205790117986
  • roc_auc_score: 0.7927754415211714

UPDATE

I tried the OVA approach and below are the results

So the high accuracy score and the high ROC value is because i oversampled the data and that artificially boosted the accuracy score.

in practice, the real accuracy for class 1 (which was 1 in 11 occurrence), was as below:

              precision    recall  f1-score   support

         0.0       1.00      0.73      0.84    125570
         1.0       0.25      0.98      0.40     11977

    accuracy                           0.75    137547
   macro avg       0.63      0.85      0.62    137547
weighted avg       0.93      0.75      0.80    137547

I think this extreme imbalance was the reason for the original problem as well

How do i fix this??

$\endgroup$

2 Answers 2

0
$\begingroup$

Try creating one model for one class. Basically you can call it as class detector, it works as following:

  • One model will predict class 0. One model will predict 1 and another one for class 2.

  • Then when you want to classify, you can use decision score from each model for that data and you select the class whose classifier outputs the highest score. This is called the one-versus-all (OvA) strategy (also called one-versus-the-rest).

  • Another strategy is to train a binary classifier for every pair of digits: one to distinguish class0 and class 1, another to distinguish class 0 and class 2, another for class 1 and class 2, and so on. This is called the one-versus-one (OvO) strategy. If there are N classes, you need to train N × (N – 1) / 2 classifiers. For your problem, this means training 4 binary classifiers! When you want to classify the data, you have to run the data through all 4 classifiers and see which class wins the most duels. The main advantage of OvO is that each classifier only needs to be trained on the part of the training set for the two classes that it must distinguish.

This will improve the overall model for your metrics.

$\endgroup$
4
  • $\begingroup$ Quite an expensive strategy. Ill try that..Currently training 1 model for each class. class combination process may be the next step $\endgroup$ Sep 21, 2023 at 7:20
  • $\begingroup$ True! It's resource expensive. If that's the case then probably tree based models could work better as they can handle multiclasses very well. Try Bagging or Boosting models $\endgroup$ Sep 21, 2023 at 8:28
  • $\begingroup$ Hey, Updated the original question, seems like OVA approach showed me some ground reality of my dataset....any other help? $\endgroup$ Sep 23, 2023 at 8:21
  • $\begingroup$ You might try tree based models if possible. $\endgroup$ Sep 25, 2023 at 8:00
0
$\begingroup$

Looking at the numbers:

  • Your classifier is quite good (or even excellent) on distinguishing class 0 and class 1.
  • Contrary to that, class 2 is either recognized as class 0 or class 1.

I would expect such a result if class 2 would be a mixture of class 0 and class 1. Imagine for example to take about 100 000 cases from class 0 and class 1 and label them class 2. Of course, the effect will probably more subtle than that.

Before tweeking with the classifier, I strongly suggest to look at the data and understand what is going there. There might be some problem with the data collection itself.

Also, a good start might be a projection to the 2-dimensional plane to have a look at the distributions. Train t-SNE or umap on all your training data and use a scatter-plot (with colors for the classes) to have a look at the data. I would expect that class 2 would mix in with class 0 and 1, while both of these classes have little intersection.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.