2
$\begingroup$

Dense and TimeDistributedDense, which one is suitable after LSTM layer in Keras?

For example,

input = Input(shape=(12,N_indepen), dtype='float32', name='inci_input')
lstm = Bidirectional(LSTM(100, activation='elu',return_sequences=False))(input)

# Dense? or TimeDistributedDense?
dense = layers.Dense(30, activation='elu')(lstm)
dense = layers.TimeDistributedDense(30, activation='elu')(lstm)

m_y = layers.Dense(1, activation='linear')(dense)

I read some articles, most of them used Dense, but some used TimeDistributedDense. I hope to know what is proper theoretically.

$\endgroup$
0
$\begingroup$

When using the TimeDistributed, you need to have a sequence through time so that you can apply the same layer (in this case, Dense), to each time slice. In your code, you have return_sequences=False which does not produce a sequence, so you need to apply Dense.

Assuming you meant to put return_sequences=True, then Dense and TimeDistributedDense do the same task in this case.

$\endgroup$
0
$\begingroup$

A dense layer will output a fixed-sized vector. This will be what you want for a classification task for example, say sentiment classification.

A TimeDistributedDense will apply a dense layer to each output of the sequence. So let's say you have a text input, represented as a sequence of word embeddings, you would apply an LSTM cell and then the same dense layer to each step output of the LSTM. This will be used for part-of-speech tagging for example, in which case you are interested in labelling each word of the sequential input with a tag.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.