LSTM with linear activation function

Question

I'm trying to do multi-step regression and I use an output layer:

LSTM(1, activation='linear', return_sequences=True)

Is this the wrong way of achieving this? Should I use a TimeDistributed(Dense(1)) as output? If yes, why?

Answer 1

I don't see any particular advantage in using linear (i.e.: none) activation. The power of Neural Network lies in their ability to "learn" non-linear patterns in your data. Moreover, the Tanh and sigmoid gates are thought to control for the stream of information that unrolls through time, they have been designed for that, and personally I'd be cautious in changing that.

If you want to do some multistep regression you have many options, but the most advanced architecture is the sequence-to-sequence, or seq2seq:

from tensorflow.keras import Sequential

from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.layers import RepeatVector, TimeDistributed
from tensorflow.keras.layers import Dense

from tensorflow.keras.activations import elu, relu

seq2seq = Sequential([

    Bidirectional(LSTM(len_input), input_shape = (len_input, no_vars)),

    RepeatVector(len_input),

    Bidirectional(LSTM(len_input, return_sequences = True)),

    TimeDistributed(Dense(hidden_size, activation = elu)),

    TimeDistributed(Dense(1, activation = relu))

])

where len_input is the length of the input sequence. The output is the same sequence shifted forward a number of steps equivalent to the lenght of your prediction. hidden_size is the size of Dense() layers, that is completely optional.

I used the Bidirectional() wrapper, but this is optional too. Not all tasks require bi-LSTM, feel free to remove it if you need.

The (combined) role of RepeatVector() and TimeDistributed() layers is to replicate the latent representation and the following Neural Network architecture for the number of steps necessary to reconstruct the output sequence. RepeatVector() generates this "multiplication", while TimeDistributed() repeats the application of on each of these repeated signals, generating the final sequence in practice.

Both input and output must be 3-dimensional numpy arrays of shape:

( number of observations , length of input sequence , number of variables )

Seq2seq models are harder to train (higher number of parameters, longer training times), but their performance is superior to other RNN architectures. At the present time, any state-of-the-art RNN is some kind of seq2seq.