1
$\begingroup$

Beginner's question regarding sequences in neural networks: suppose I have classification problem that looks like:
X = very long sequence of varying length. Y = class (assume for simplicity y=0/1).
The class depends on the whole sequence - it is impossible to determine the class without seeing all of it.
How NN handle this? I have read various methods, such as truncating the sequence (which is a problem - there is no class for truncated part of the sequence, so I cannot label it), using encoder/decoder lstm (again, couldn't completely understand how it solves the problem), etc.

$\endgroup$
2
$\begingroup$

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do :

  • Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the Input layer : the 0s will not be taken into account for training the model. Bellow an example using the Keras framework (not complete, not tested) :

    from keras.models import Model
    from keras.layers import Input, LSTM, Dense
    input = Input(shape=[None, nb_of_features_per_timestep])
    layer = Masking(mask_value=0)(input)
    layer = LSTM(50, return_sequences=False)(layer)
    output = Dense(nb_classe)(layer)
    model = Model(input,output)
    
  • You can also use the bucketing trick : you group your dataset into "buckets" of same size (approximately). You can then create batches of same size. You don't need the Masking layer anymore.

  • You can also use fully convolutional network. Those networks can also take Input with different size. I let you search if you want to use that.

Concerning encoder/decoder : those model can maybe help you but I think that you will have to use one of the tricks that I mentioned. If you don't understand how RNNs work, I recommend you to dig a little into the subject before.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.