I've already seen several similair questions but I did not understand anything, what is the interest of TimeDistributed? why we need to insert a TimeDistributed layer after LSTM to establish the time series prediction


Ok let's say you have an LSTM() layer with return_sequences = True set. That means each LSTM cell in it is outputting its value. The output of the layer is therefore a sequence of outputs, not just the final one. That means the output is a processed time series, with sequential information in it.

You want Dense() layers to take this information and use it to process the signal further. In particular, you want some Dense architecture to process each of these sequential outputs one by one. By using the TimeDistributed() wrapper it's like if you're iterating the same Dense() architecture on this sequential output.

I typically use TimeDistributed(Dense()) at the bottom of a seq2seq Decoder, in order produce the final output sequence.

  • $\begingroup$ This is the clearest explanation so far on the matter. Thank you very much. Can you please give an example what will happen if we apply, let us say TimeDis(Dense([100, 4, 16])) ? where [100, 4, 16] is the output of the last LSTM layer I have in my model? $\endgroup$
    – Avra
    Sep 19 at 13:20

I think you mean TimeDistributedDense? Anyway this should clear up you understanding in one way or other.

TimeDistributedDense applies a same Dense (fully-connected) operation to every timestep of a 3D tensor.

The most common scenario for using TimeDistributedDense is using a recurrent NN for tagging task.e.g. POS labeling or slot filling task.

In this kind of task: For each sample, the input is a sequence ($a_1,a_2,a_3,a_4...a_N$) and the output is a sequence ($b_1,b_2,b_3,b_4...b_N$) with the same length. $b_i$ could be viewed as the label of $a_i$. Push $a_1$ into a recurrent neural net to get output $b_1$. Than push $a_2$ and the hidden output of $a_1$ to get $b_2$...

If you want to model this by Keras, you just need to used a TimeDistributedDense after a RNN or LSTM layer(with return_sequence=True) to make the cost function is calculated on all time-step output. If you don't use TimeDistributedDense ans set the return_sequence of RNN=False, then the cost is calculated on the last time-step output and you could only get the last bN.

Also, here is a link for more info -> https://github.com/keras-team/keras/issues/1029


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.