What's the difference of stateless LSTM and a normal feed-forward NN?

Question

From what I understand, the whole point of LSTM is for the network to establish long-term dependencies in the data, i.e. an event happening now may be in some way determined by something that happened sometime in the past, and that may not be in the batch of data currently being presented, but many sequences or batches previously.

So I understand that a stateful LSTM does this, and the network will retain the state until reset_states() is called. But with stateless LSTM, where the state isn't retained between batches, then how does that differ from a normal feed-forward perceptron network?

I'm assuming that even though in stateless LSTM you don't control the states as you would in a stateful model, that some state is still retained at passed on between sequences? Is it passed on between batches?

I can understand why someone would use stateful LSTM, but why would someone use stateless LSTM instead of a feed-forward perceptron network? Or vice-versa, why would someone use a regular FF instead of a stateless LSTM?

Answer 1

Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in turn it is vastly more performant, while a statefull LSTM you can have varying timesteps, but at a performance penalty.

The stateless LSTM does have state, it's just implemented differently. Instead of managing it yourself by constantly calling reset_states() your data is structure in such a way that it is automatically reset when the end of one series of timesteps is done.

Let's create an explicit example. Say you have some label data y[i] where i is the index, and input data x[i, t] where i corresponds to your label's index, and t is at what timestep this was observed. (The x and ys can be vectors, matrices, or any type of tensor really.)

Say you have 4 output labels (not 4 classes), and 2 to 4 timesteps for each of them. Your data might look like this:

                   x[0, 0], x[0, 1]  ->  y[0]
          x[1, 0], x[1, 1], x[1, 2]  ->  y[1]
 x[2, 0], x[2, 1], x[2, 2], x[2, 3]  ->  y[2]
                   x[3, 0], x[3, 1]  ->  y[3]

This is the kind of data a statefull LSTM can handle. But at the end of each label, you would have to call reset_states() to indicate that your no longer dependent on the previous series of xs. This is slow.

With a stateless LSTM you would pad the x so that you have a rectangular matrix:

 0        0        x[0, 0], x[0, 1]  ->  y[0]
 0        x[1, 0], x[1, 1], x[1, 2]  ->  y[1]
 x[2, 0], x[2, 1], x[2, 2], x[2, 3]  ->  y[2]
 0        0        x[3, 0], x[3, 1]  ->  y[3]

In this case, there is an implicit reset of state for each label. This way of doing it is vastly more performant. But there might occur issues with bogging down your data with empty data. Personally I tend to add a binary flag to each x vector to indicate to the network in the LSTM that it should probably ignore the data.

I'm not sure about the specifics of how stateless LSTMs are implemented, but I think it might be about unrolling the LSTM units.

Ultimately, both stateless and statefull LSTM's do the same thing, they are just implemented differently.

why would someone use stateless LSTM instead of a feed-forward perceptron network?

Well, ironically it's because stateless LSTM have state, which means it can take into account the previous timestep input values when doing predictions.