2
$\begingroup$

I've already seen several similair questions but I did not understand anything, what is the interest of TimeDistributed? why we need to insert a TimeDistributed layer after LSTM to establish the time series prediction

$\endgroup$
1
$\begingroup$

Ok let's say you have an LSTM() layer with return_sequences = True set. That means each LSTM cell in it is outputting its value. The output of the layer is therefore a sequence of outputs, not just the final one. That means the output is a processed time series, with sequential information in it.

You want Dense() layers to take this information and use it to process the signal further. In particular, you want some Dense architecture to process each of these sequential outputs one by one. By using the TimeDistributed() wrapper it's like if you're iterating the same Dense() architecture on this sequential output.

I typically use TimeDistributed(Dense()) at the bottom of a seq2seq Decoder, in order produce the final output sequence.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I think you mean TimeDistributedDense? Anyway this should clear up you understanding in one way or other.

TimeDistributedDense applies a same Dense (fully-connected) operation to every timestep of a 3D tensor.

The most common scenario for using TimeDistributedDense is using a recurrent NN for tagging task.e.g. POS labeling or slot filling task.

In this kind of task: For each sample, the input is a sequence ($a_1,a_2,a_3,a_4...a_N$) and the output is a sequence ($b_1,b_2,b_3,b_4...b_N$) with the same length. $b_i$ could be viewed as the label of $a_i$. Push $a_1$ into a recurrent neural net to get output $b_1$. Than push $a_2$ and the hidden output of $a_1$ to get $b_2$...

If you want to model this by Keras, you just need to used a TimeDistributedDense after a RNN or LSTM layer(with return_sequence=True) to make the cost function is calculated on all time-step output. If you don't use TimeDistributedDense ans set the return_sequence of RNN=False, then the cost is calculated on the last time-step output and you could only get the last bN.

Also, here is a link for more info -> https://github.com/keras-team/keras/issues/1029

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.