1
$\begingroup$

I am working on multi-class problem with sequences. My dataset is composed of sequences of data with different length.

E.g. 1500 labeled samples: 500 datapoint belongs to class A, 500 class B and 500 class C. For A and B the sequence length is 400 and 1000 respectively, and for class C the sequence length is 100.

In order to train the model I have applied post-padding on the sequences so that all the sequences have the same length. The resulting dataset has this shape (1500,1000).

I have tried first EMBEDDING+LSTM (mask_zero=True) to map and classify the sequences but even if the model achieve very high accuracy, evaluate the model with random/fake data it will classify based on the sequences' length: suggesting that the model is learning on the lengths instead of values.

The main problem is that the model is much more learning on 0s even if we use "mask_zero" into the embedding layer. My question is:

Does someone can suggest an approach to deal with very long sequences? Considering that we have very short and very long sequences to predict?

I am exploring another different approach:

  1. Train an Autoencoder (ANN or 1DCNN) to reduce the sequence length. Use the encoder and train again the Embedding layer + LSTM layer.

Thanks.

Improve this question
$\endgroup$
8
  • $\begingroup$ What kind of data is it? Text or numbers? Are the data points related to each other to try a compression method? $\endgroup$
    – Nicolas Martin
    Jul 26, 2021 at 18:06
  • $\begingroup$ Sequences are numbers. Each elements in a single sequence are related . what's the compression you suggests? $\endgroup$
    – scalessio
    Jul 29, 2021 at 10:33
  • $\begingroup$ If you use a logarithm, you can compress float data efficiently, but you might loss some information. For instance: log(20 000 000) = 7.03 and log(2 000) = 3.30. I used to apply this solution in similar cases than you have, and the results were good. $\endgroup$
    – Nicolas Martin
    Jul 29, 2021 at 13:41
  • $\begingroup$ how exactly should I apply this methodology? my dataset consists in a sequences of fixed length. if I understood I should apply the log to the length? $\endgroup$
    – scalessio
    Jul 29, 2021 at 15:02
  • $\begingroup$ Can you give an example of the values of a sequence please? $\endgroup$
    – Nicolas Martin
    Jul 29, 2021 at 15:17

1 Answer 1

Reset to default
0
$\begingroup$

Try the following:

  1. Check if text preprocessing is done correctly.
  2. With different sequence lengths (Ex: 1000, 750, 500 etc.)
  3. After the LSTM layer, try adding the ANN layers and check the results.
  4. Check the Transformers architecture for text classification. Reference: https://ai-brewery.medium.com/simple-chatbot-using-bert-and-pytorch-part-1-2735643e0baa
Improve this answer
$\endgroup$
1
  • $\begingroup$ hi thanks for your reply. I don't need text processing since the sequences are already floating points values. And yes I have tried an architecture with embedding layer +lstm +ANN but it doesn't work: with random data the model predicts the classes based on sequence's length. $\endgroup$
    – scalessio
    Jul 29, 2021 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.