1
$\begingroup$

I'm using LSTM for time series prediction, my data is highly skewed, with class weight 197.16865807 : 0.50127117

With Label 0 : 25359 and Label 1 : 9974641

my model is shown below

n_input = 100
n_features = 36
class_weights = class_weight.compute_class_weight('balanced',
                                              np.unique(y_target),
                                              y_target)
model = tf.keras.Sequential([
        tf.keras.layers.LSTM(64, activation='tanh', input_shape=(n_input, n_features),return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(1,activation='sigmoid')])

model.compile(optimizer='adam', loss= 'binary_crossentropy' ,metrics=METRICS)
model.fit_generator(train_generator, epochs= 1,steps_per_epoch=len(train_generator),class_weight=class_weight)

I have tried the following method to dealing with my unchanged accuracy and loss value.

  1. I tried to adjust class weight, but it seems not working, my predictions are all 1
  2. I tried to use RandomSampler from imblearn to undersample my data, but the accuracy stucked at 50%
  3. I tried to change the loss function to weighted_cross_entropy_with_logits, but I did not find any examples show how to use it in Sequential model like the one above

I feel my model is not predict the result, since when I feed balanced dataset, the accuracy is around 50%, when I feed imbalanced dataset, the accuracy is 99%.

Can anyone help me with this? I wondering if it's the problem of my model, or the problem of my imbalanced dataset

Thank you!

$\endgroup$
1
$\begingroup$

The dataset contains ~25K class '0' samples and ~10M class '1' sample. This clearly tells us that the LSTM would be learning patterns more popular in class '1' instances. Here are some improvements you can try:

  1. Instead of undersampling the class '1' labels, oversample the number of instances of class '0'.

  2. Accuracy is not a very good metric in cases of unbalanced datasets. Use 'Macro-F score to evaluate the performance of your model.

  3. Use tensorboard to see how the weights and the gradients of each layer are changing.

| improve this answer | |
$\endgroup$
0
$\begingroup$

There's a few options I would try:

Firstly, remove the class weight. See if it helps you get higher than 50% on a balanced dataset.

Secondly, either oversample Label 0 or undersample Label 1 instead of using the class weight.

Thirdly, try using focal loss as the loss function so your gradient updates are more focussed on the examples you are getting wrong.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.