For an instance, we have M matrix of size n*k, I would like to treat each matrix independently and perform the convolution operation at each one in order to introduce to an LSTM layer the outputs of the convolution layer. can I treat each matrix independently or do I have to use the channel technique and how I can provide each LSTM cell? you find the architecture link below, thank you in advance!!
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I write an answer here since I don't have enough reputation to write a comment.
What about treating your M matrices as sequence, feeding them to a TimeDistributed(Conv2D) layer(s) and then to an LSTM layer? In this way the Conv layer should extract feature maps that are treated by the LSTM in a sequence-like fashion, updating/accumulating the states up to the M matrix. Afterwards you should put a Dense layer with sigmoid/softmax activation and get class probabilities.
$\begingroup$
$\endgroup$
First of all, thank you for your answer, I think your idea is the ideal one for my problem, you find the architecture update as well as a piece of code that represents the construction of the architecture graph, please take a look and give your opinion.
M = 4 # number of matrix per sample
n = 50 # number of rows for each matrix
k = 50 # number of columns for each matrix
A1 = Input(shape=(M,n,k,1),name='A1') # 1 represents the number of channels
A2 = TimeDistributed(Conv2D(64,2,data_format='channels_last'), input_shape=(M,n,k,1))(A1)
A3 = TimeDistributed(Flatten())(A2)
A4 = LSTM(8)(A3)
A5 = Dense(12)(A4)
output_layer = Dense(1, activation='softmax', name='softmax_output')(A5)
merged = Model(inputs=[A1],outputs=output_layer)
plot_model(merged,to_file='graph_cnn_rnn.png',show_shapes=True)