Dense? or TimeDistributedDense? after LSTM layer in Keras

Question

Dense and TimeDistributedDense, which one is suitable after LSTM layer in Keras?

For example,

input = Input(shape=(12,N_indepen), dtype='float32', name='inci_input')
lstm = Bidirectional(LSTM(100, activation='elu',return_sequences=False))(input)

# Dense? or TimeDistributedDense?
dense = layers.Dense(30, activation='elu')(lstm)
dense = layers.TimeDistributedDense(30, activation='elu')(lstm)

m_y = layers.Dense(1, activation='linear')(dense)

I read some articles, most of them used Dense, but some used TimeDistributedDense. I hope to know what is proper theoretically.

Answer 1

When using the TimeDistributed, you need to have a sequence through time so that you can apply the same layer (in this case, Dense), to each time slice. In your code, you have return_sequences=False which does not produce a sequence, so you need to apply Dense.

Assuming you meant to put return_sequences=True, then Dense and TimeDistributedDense do the same task in this case.

Answer 2

A dense layer will output a fixed-sized vector. This will be what you want for a classification task for example, say sentiment classification.

A TimeDistributedDense will apply a dense layer to each output of the sequence. So let's say you have a text input, represented as a sequence of word embeddings, you would apply an LSTM cell and then the same dense layer to each step output of the LSTM. This will be used for part-of-speech tagging for example, in which case you are interested in labelling each word of the sequential input with a tag.