I am working on multi-class problem with sequences. My dataset is composed of sequences of data with different length.
E.g. 1500 labeled samples: 500 datapoint belongs to class A, 500 class B and 500 class C. For A and B the sequence length is 400 and 1000 respectively, and for class C the sequence length is 100.
In order to train the model I have applied post-padding on the sequences so that all the sequences have the same length. The resulting dataset has this shape (1500,1000).
I have tried first EMBEDDING+LSTM (mask_zero=True) to map and classify the sequences but even if the model achieve very high accuracy, evaluate the model with random/fake data it will classify based on the sequences' length: suggesting that the model is learning on the lengths instead of values.
The main problem is that the model is much more learning on 0s even if we use "mask_zero" into the embedding layer. My question is:
Does someone can suggest an approach to deal with very long sequences? Considering that we have very short and very long sequences to predict?
I am exploring another different approach:
- Train an Autoencoder (ANN or 1DCNN) to reduce the sequence length. Use the encoder and train again the Embedding layer + LSTM layer.