I've implemented an LSTM auto-encoder. It trained well, and does what I want it to so far. But, I think I've misunderstood something fundamental about lstms.

In a simple dense network whose input layer is size 7, I can feed a tensor of shape (batch_size, 7) where batch_size = len(data) / 7 and get the expected results. I successfully perform inference on the entire dataset at once.

I thought it would be the same with my LSTM autoencoder, but it only seems to work if I feed the data one entry at a time in a for-loop.

Dense reconstruction using of for-loop: OR a single "batch":

enter image description here

LSTM reconstruction using for-loop: enter image description here

LSTM using a single "batch": enter image description here

I know that LSTMs are sequential by nature, but I was under the impression that you can feed batches to the LSTM and some fancy optimization of for-loops occurs. Did I miss something conceptually?

  • $\begingroup$ LSTMs certainly accept batches as input and it is common to do so. Maybe you should check the order of dimensions is the correct one, as it is frequent for LSTMs to receive their input as a tensor of shape [timesteps, batch, feature] by default. $\endgroup$
    – noe
    Commented Feb 1, 2021 at 15:56
  • $\begingroup$ @noe, yes that's true. In my case, I'm using batch_first=True. I have verified this previously. The implementation is a custom use of single LSTM cells rather than a full LSTM layer. I wonder if that plays a part? $\endgroup$ Commented Feb 1, 2021 at 20:51


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.