What is the job of "RepeatVector" and "TimeDistributed"?

Question

I read about them in Keras documentation and other websites, but I couldn't exactly understand what exactly they do and how should we use them in designing many-to-many or encoder-decoder LSTM networks?

I saw them used in the solution of this problem here.

model = Sequential()  
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))  
model.add(RepeatVector(10))
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))  
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear'))   
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])

Answer 1

tf.keras.layers.RepeatVector

According to the docs :

Repeats the input n times.

They have also provided an example :

model = Sequential()
model.add(Dense(32, input_dim=32))
# now: model.output_shape == (None, 32)
# note: `None` is the batch dimension
model.add(RepeatVector(3))
# now: model.output_shape == (None, 3, 32)

In the above example, the RepeatVector layer repeats the incoming inputs a specific number of time. The shape of the input in the above example was ( 32 , ). But the output shape of the RepeatVector was ( 3 , 32 ), since the inputs were repeated 3 times.

tf.keras.layers.TimeDistributed()

According to the docs :

This wrapper allows to apply a layer to every temporal slice of an input. The input should be at least 3D, and the dimension of index one will be considered to be the temporal dimension.

You can refer to the example at their website.

TimeDistributed layer applies a specific layer such as Dense to every sample it receives as an input. Suppose the input size is ( 13 , 10 , 6 ). Now, I need to apply a Dense layer to every slice of shape ( 10 , 6 ). Then I would wrap the Dense layer in a TimeDistributed layer.

model.add( TimeDistributed( Dense( 12 , input_shape=( 10 , 6 ) )) )

The output shape of such a layer would be ( 13 , 10 , 12 ). Hence, the operation of the Dense layer was applied to each temporal slice as mentioned.

Answer 2

For encoder-decoder, your input is squashed into a single feature vector, if you want your output to regenerate the same dimension as the original input, you are "artificially" converting this feature tensor from 1D into 2D by replicating it using RepeatVector().