# How to feed multiple asymmetric inputs to LSTM layer?

I'm trying to create an encoder-decoder architecture with an LSTM encoder. The intention is to use both the input image as well as the class label as inputs to the encoder, and to have them share the LSTM layers. Before working on my actual dataset, I want to get it working on MNIST just to see the architecture is set up right, so I'm using and input image of size (28, 28, 1) and a one-hot encoded class vector of size (10,).

The keras documentation presents multiple inputs as follows:

    main_input = Input(shape=(100,), dtype='int32', name='main_input')
x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)
lstm_out = LSTM(32)(x)
auxiliary_input = Input(shape=(5,), name='aux_input')
x = keras.layers.concatenate([lstm_out, auxiliary_input])

x = Dense(64, activation='relu')(x)


However, this requires the two concatenated tensors to have an identical shape apart from the concatenation axis, which is not the case in my situation.

Here is my encoder so far - it obviously doesn't work for the reason mentioned above. In the latest iteration I attempted to flatten the input image and concatenate the 2d tensors. This works, but runs into the problem of LSTM requiring 3d input.

    """ LSTM encoder with 512 cells, softmax activation for signal classification,
linear activation for extracting latent features"""
input_img = Input(shape=self.img_shape)
input_img = Flatten()(input_img)
input_class = Input(shape=(self.num_classes,))

merged = Concatenate()([input_img, input_class])
encoder = LSTM(512, activation=LeakyReLU(alpha=0.2), return_sequences=True)(merged)
encoder = LSTM(512, activation=LeakyReLU(alpha=0.2))(encoder)

features = Dense(self.latent_dim, activation='linear')(encoder)
classes = Dense(self.num_classes, activation='softmax')(encoder)

return Model([input_img, input_class], [classes, features])


To be clear, I know I could just make separate LSTM layers for both inputs. I do not want to do that, I want them to share the layers.