LSTM model is producing really bad results for multiclass text classification for imbalanced small dataset

Question

I am training a LSTM model on my current dataset to predict the multiclass categories - there are 18 mutually exclusive categories and the dataset has ~ 500 rows only (a really small dataset). I am handling the class imbalance using the following:

from sklearn.utils import class_weight
class_weights = list(class_weight.compute_class_weight('balanced',
                                               classes = np.unique(df['categories']),
                                               y = df['categories']))
weights = {}
for index, weight in enumerate(class_weights):
  weights[index] = weight

Post this I am building my LSTM model and have been evaluating this model using PRC in tf.metrics as this is an imbalanced target classification problem

METRICS = [ tf.metrics.AUC(name='prc', curve='PR'), # precision-recall curve]
model = Sequential()
model.add(Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=X.shape[1]))
model.add(SpatialDropout1D(0.2))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(18, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam',  metrics=METRICS)
print(model.summary())

and finally:

  history = model.fit(X_train,
            y_train,
            batch_size=10,
            epochs=10,
            verbose=1,
            class_weight=weights,
            validation_data=(X_test,y_test))

Now when I look at the results, the training prc is coming out to be really high whereas my val_prc is really low. An example with 10 epochs:

 Epoch 1/10
 30/30 [==============================] - 5s 174ms/step - loss: 2.9951 - prc: 0.0682 - 
 val_loss: 2.8865 - val_prc: 0.0639
 Epoch 2/10
 30/30 [==============================] - 5s 169ms/step - loss: 2.9556 - prc: 0.0993 - 
 val_loss: 2.8901 - val_prc: 0.0523
 .....
 Epoch 8/10
 30/30 [==============================] - 6s 189ms/step - loss: 1.2494 - prc: 0.6415 - 
 val_loss: 3.0662 - val_prc: 0.0728
 Epoch 9/10
 30/30 [==============================] - 6s 210ms/step - loss: 0.9237 - prc: 0.8302 - 
 val_loss: 3.0624 - val_prc: 0.1006
 Epoch 10/10
 30/30 [==============================] - 6s 184ms/step - loss: 0.7452 - prc: 0.9017 - 
 val_loss: 3.5035 - val_prc: 0.0821

My questions are:

1. Is the evaluation metric correct that I am using considering it is an imbalanced class problem?
2. Am I treating the imbalance correctly with the code that I have written in the first place and most importantly, am I using this correct in the model.fit() ?

How can I resolve this? Is there any alternative approach that you can suggest?

Answer 1

It's good that you are using class weights to handle the class imbalance in your dataset. Class weights can help the model to learn from underrepresented classes better. However, you should also consider using other techniques to handle the class imbalance, such as undersampling the majority classes or oversampling the minority classes.

Using the precision-recall curve (PRC) as the evaluation metric is a good choice for imbalanced classification problems, because it can give you a more complete picture of the model's performance on different classes. However, you should also consider using other metrics, such as the F1 score or the weighted average of the precision and recall scores, to get a more comprehensive view of the model's performance.

It's also worth noting that the low validation PRC score may be due to the small size of your dataset. With only 500 rows, it may be difficult for the model to generalize well to unseen data. One way to improve the model's performance would be to gather more data, if possible.

Another thing you could try is using different model architectures or optimization techniques to see if you can improve the model's performance. You might also want to try different preprocessing techniques, such as removing stop words or using different word embeddings, to see if they have an impact on the model's performance.