0
$\begingroup$

I'm implementing a LSTM model with Keras. My dataset is composed by words and each word is an 837 long vector. I grouped the words in groups of 20 and to do this I padded them: initially I had groups of words of variable length and the maximum group length that I found was 20, this is why I padded all groups to 20.

For example, a group of 5 words is:

[[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837]]

where xi is the i-th feature of the vector.

To pad this group to a length of 20, I added 15 vectors composed by 837 feature with value equal to zeros:

[[0.......0],
............
............
[0........0]]

So, at the end, my group is of the form:

[[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[0...........0],
..............
..............
[0...........0]]

How could I ignore the vectors of zeros during training?

$\endgroup$
1
$\begingroup$

You can use Masking layer (with mask value of zero) before LSTM layer in order to ignore all timesteps with only zeros (i.e. zeros vector). You can find more information about this layer on its documentation. Here is an example from documentation which uses Masking layer:

Consider a Numpy data array x of shape (samples, timesteps, features), to be fed to an LSTM layer. You want to mask sample #0 at timestep #3, and sample #2 at timestep #5, because you lack features for these sample timesteps. You can do:

  • set x[0, 3, :] = 0. and x[2, 5, :] = 0.
  • insert a Masking layer with mask_value=0. before the LSTM layer:

    model = Sequential()
    model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
    model.add(LSTM(32))
    
$\endgroup$
1
  • $\begingroup$ Thank you, I'll try. $\endgroup$
    – pairon
    Apr 10 '20 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.