For an instance, we have M matrix of size n*k, I would like to treat each matrix independently and perform the convolution operation at each one in order to introduce to an LSTM layer the outputs of the convolution layer. can I treat each matrix independently or do I have to use the channel technique and how I can provide each LSTM cell? you find the architecture link below, thank you in advance!! r


2 Answers 2


I write an answer here since I don't have enough reputation to write a comment.

What about treating your M matrices as sequence, feeding them to a TimeDistributed(Conv2D) layer(s) and then to an LSTM layer? In this way the Conv layer should extract feature maps that are treated by the LSTM in a sequence-like fashion, updating/accumulating the states up to the M matrix. Afterwards you should put a Dense layer with sigmoid/softmax activation and get class probabilities.


First of all, thank you for your answer, I think your idea is the ideal one for my problem, you find the architecture update as well as a piece of code that represents the construction of the architecture graph, please take a look and give your opinion.enter image description here

M = 4  # number of matrix per sample
n = 50 # number of rows for each matrix
k = 50 # number of columns for each matrix

A1 = Input(shape=(M,n,k,1),name='A1') # 1 represents the number of channels
A2 = TimeDistributed(Conv2D(64,2,data_format='channels_last'), input_shape=(M,n,k,1))(A1)  
A3 = TimeDistributed(Flatten())(A2)
A4 = LSTM(8)(A3)
A5 = Dense(12)(A4)
output_layer = Dense(1, activation='softmax', name='softmax_output')(A5)
merged = Model(inputs=[A1],outputs=output_layer)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.