LSTM loss value not change, accuracy stucked at 50%

Question

I'm using LSTM for time series prediction, my data is highly skewed, with class weight 197.16865807 : 0.50127117

With Label 0 : 25359 and Label 1 : 9974641

my model is shown below

n_input = 100
n_features = 36
class_weights = class_weight.compute_class_weight('balanced',
                                              np.unique(y_target),
                                              y_target)
model = tf.keras.Sequential([
        tf.keras.layers.LSTM(64, activation='tanh', input_shape=(n_input, n_features),return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(1,activation='sigmoid')])

model.compile(optimizer='adam', loss= 'binary_crossentropy' ,metrics=METRICS)
model.fit_generator(train_generator, epochs= 1,steps_per_epoch=len(train_generator),class_weight=class_weight)

I have tried the following method to dealing with my unchanged accuracy and loss value.

1. I tried to adjust class weight, but it seems not working, my predictions are all 1
2. I tried to use RandomSampler from imblearn to undersample my data, but the accuracy stucked at 50%
3. I tried to change the loss function to weighted_cross_entropy_with_logits, but I did not find any examples show how to use it in Sequential model like the one above

I feel my model is not predict the result, since when I feed balanced dataset, the accuracy is around 50%, when I feed imbalanced dataset, the accuracy is 99%.

Can anyone help me with this? I wondering if it's the problem of my model, or the problem of my imbalanced dataset

Thank you!

Answer 1

The dataset contains ~25K class '0' samples and ~10M class '1' sample. This clearly tells us that the LSTM would be learning patterns more popular in class '1' instances. Here are some improvements you can try:

1. Instead of undersampling the class '1' labels, oversample the number of instances of class '0'.

2. Accuracy is not a very good metric in cases of unbalanced datasets. Use 'Macro-F score to evaluate the performance of your model.

3. Use tensorboard to see how the weights and the gradients of each layer are changing.

Answer 2

There's a few options I would try:

Firstly, remove the class weight. See if it helps you get higher than 50% on a balanced dataset.

Secondly, either oversample Label 0 or undersample Label 1 instead of using the class weight.

Thirdly, try using focal loss as the loss function so your gradient updates are more focussed on the examples you are getting wrong.