I'm trying to do multi-step regression and I use an output layer:
LSTM(1, activation='linear', return_sequences=True)
Is this the wrong way of achieving this? Should I use a TimeDistributed(Dense(1)) as output? If yes, why?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
I don't see any particular advantage in using linear (i.e.: none) activation. The power of Neural Network lies in their ability to "learn" non-linear patterns in your data. Moreover, the Tanh and sigmoid gates are thought to control for the stream of information that unrolls through time, they have been designed for that, and personally I'd be cautious in changing that.
If you want to do some multistep regression you have many options, but the most advanced architecture is the sequence-to-sequence, or seq2seq:
from tensorflow.keras import Sequential from tensorflow.keras.layers import Bidirectional, LSTM from tensorflow.keras.layers import RepeatVector, TimeDistributed from tensorflow.keras.layers import Dense from tensorflow.keras.activations import elu, relu seq2seq = Sequential([ Bidirectional(LSTM(len_input), input_shape = (len_input, no_vars)), RepeatVector(len_input), Bidirectional(LSTM(len_input, return_sequences = True)), TimeDistributed(Dense(hidden_size, activation = elu)), TimeDistributed(Dense(1, activation = relu)) ])
len_input is the length of the input sequence. The output is the same sequence shifted forward a number of steps equivalent to the lenght of your prediction.
hidden_size is the size of
Dense() layers, that is completely optional.
I used the
Bidirectional() wrapper, but this is optional too. Not all tasks require bi-LSTM, feel free to remove it if you need.
The (combined) role of
TimeDistributed() layers is to replicate the latent representation and the following Neural Network architecture for the number of steps necessary to reconstruct the output sequence.
RepeatVector() generates this "multiplication", while
TimeDistributed() repeats the application of on each of these repeated signals, generating the final sequence in practice.
Both input and output must be 3-dimensional numpy arrays of shape:
( number of observations , length of input sequence , number of variables )
Seq2seq models are harder to train (higher number of parameters, longer training times), but their performance is superior to other RNN architectures. At the present time, any state-of-the-art RNN is some kind of seq2seq.