Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in turn it is vastly more performant, while a statefull LSTM you can have varying timesteps, but at a performance penalty.
The stateless LSTM does have state, it's just implemented differently. Instead of managing it yourself by constantly calling reset_states()
your data is structure in such a way that it is automatically reset when the end of one series of timesteps is done.
Let's create an explicit example. Say you have some label data y[i]
where i
is the index, and input data x[i, t]
where i
corresponds to your label's index, and t
is at what timestep this was observed. (The x
and y
s can be vectors, matrices, or any type of tensor really.)
Say you have 4 output labels (not 4 classes), and 2 to 4 timesteps for each of them. Your data might look like this:
x[0, 0], x[0, 1] -> y[0]
x[1, 0], x[1, 1], x[1, 2] -> y[1]
x[2, 0], x[2, 1], x[2, 2], x[2, 3] -> y[2]
x[3, 0], x[3, 1] -> y[3]
This is the kind of data a statefull LSTM can handle. But at the end of each label, you would have to call reset_states()
to indicate that your no longer dependent on the previous series of x
s. This is slow.
With a stateless LSTM you would pad the x
so that you have a rectangular matrix:
0 0 x[0, 0], x[0, 1] -> y[0]
0 x[1, 0], x[1, 1], x[1, 2] -> y[1]
x[2, 0], x[2, 1], x[2, 2], x[2, 3] -> y[2]
0 0 x[3, 0], x[3, 1] -> y[3]
In this case, there is an implicit reset of state for each label. This way of doing it is vastly more performant. But there might occur issues with bogging down your data with empty data. Personally I tend to add a binary flag to each x
vector to indicate to the network in the LSTM that it should probably ignore the data.
I'm not sure about the specifics of how stateless LSTMs are implemented, but I think it might be about unrolling the LSTM units.
Ultimately, both stateless and statefull LSTM's do the same thing, they are just implemented differently.
why would someone use stateless LSTM instead of a feed-forward perceptron network?
Well, ironically it's because stateless LSTM have state, which means it can take into account the previous timestep input values when doing predictions.