I'm trying to build an neural net in Keras that would look like this:

enter image description here

Where $x_1$, $x_2$, ... are input vectors that undergo the same transformation $f$. $f$ is itself a layer whose parameters must be learned. The sequence length $n$ is variable across instances.

I'm having trouble understanding two things here:

  1. What should the input look like?
    I'm thinking of a 2D tensor with shape (number_of_x_inputs, x_dimension), where x_dimension is the length of a single vector $x$. Can such 2D tensor have a variable shape? I know tensors can have variable shapes for batch processing, but I don't know if that helps me here.

  2. How do I pass each input vector through the same transformation before feeding it to the RNN layer?
    Is there a way to sort of extend for example a GRU so that an $f$ layer is added before going through the actual GRU cell?

  • $\begingroup$ Is f the same st every time step? $\endgroup$
    – kbrose
    Commented Feb 16, 2018 at 0:09
  • $\begingroup$ Yes it is. It is also optimized during training, so it changes between iterations, but for a given set of inputs x_1, ..., x_n, f is the same. $\endgroup$
    – hugom
    Commented Feb 16, 2018 at 1:50

1 Answer 1

  1. What should the input look like?

You are right to think a 2D tensor, but usually we add one more dimension for the batch. You can indeed have a variable length number_of_x_inputs, but to train during batch processing all inputs in a single batch will need to have the same shape. (Setting batch size to 1 will get around this.) During inference, you can have whatever length you want. See below code sample.

  1. How do I pass each input vector through the same transformation before feeding it to the RNN layer?

Use TimeDistributed. The example below passes all vectors $x_i$ through the same feed forward network (Dense(5, ...)), but you should be able to swap that out for whatever f you were thinking.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed

x_dimension = 16
num_classes = 2

model = Sequential()

model.add(TimeDistributed(Dense(5, activation='relu'),
          input_shape=(None, x_dimension)))
model.add(LSTM(32, return_sequences=True))
model.add(Dense(num_classes, activation='softmax'))


This prints the following model:

Layer (type)                            Output Shape                        Param #
time_distributed_1 (TimeDistributed)    (None, None, 5)                     85
lstm_1 (LSTM)                           (None, None, 32)                    4864
lstm_2 (LSTM)                           (None, 8)                           1312
dense_2 (Dense)                         (None, 2)                           18
Total params: 6,279
Trainable params: 6,279
Non-trainable params: 0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.