How to build a recurrent neural net in Keras where each input goes through a layer first?

Question

I'm trying to build an neural net in Keras that would look like this:

Where $x_1$, $x_2$, ... are input vectors that undergo the same transformation $f$. $f$ is itself a layer whose parameters must be learned. The sequence length $n$ is variable across instances.

I'm having trouble understanding two things here:

1. What should the input look like?
   I'm thinking of a 2D tensor with shape (number_of_x_inputs, x_dimension), where x_dimension is the length of a single vector $x$. Can such 2D tensor have a variable shape? I know tensors can have variable shapes for batch processing, but I don't know if that helps me here.

2. How do I pass each input vector through the same transformation before feeding it to the RNN layer?
   Is there a way to sort of extend for example a GRU so that an $f$ layer is added before going through the actual GRU cell?

Answer 1

1. What should the input look like?

You are right to think a 2D tensor, but usually we add one more dimension for the batch. You can indeed have a variable length number_of_x_inputs, but to train during batch processing all inputs in a single batch will need to have the same shape. (Setting batch size to 1 will get around this.) During inference, you can have whatever length you want. See below code sample.

2. How do I pass each input vector through the same transformation before feeding it to the RNN layer?

Use TimeDistributed. The example below passes all vectors $x_i$ through the same feed forward network (Dense(5, ...)), but you should be able to swap that out for whatever f you were thinking.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed

x_dimension = 16
num_classes = 2

model = Sequential()

model.add(TimeDistributed(Dense(5, activation='relu'),
          input_shape=(None, x_dimension)))
model.add(LSTM(32, return_sequences=True))
model.add(LSTM(8))
model.add(Dense(num_classes, activation='softmax'))

print(model.summary(90))

This prints the following model:

Layer (type)                            Output Shape                        Param #
==========================================================================================
time_distributed_1 (TimeDistributed)    (None, None, 5)                     85
__________________________________________________________________________________________
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, 8)                           1312
__________________________________________________________________________________________
dense_2 (Dense)                         (None, 2)                           18
==========================================================================================
Total params: 6,279
Trainable params: 6,279
Non-trainable params: 0
__________________________________________________________________________________________