0
$\begingroup$

If I use stacked CNN layers with 3x3 kernels, zero padding, and with no pooling layers, the output feature map will consist of feature vectors, each vector of which is related directly to the original 3x3 block of the input image, right?

Therefore, for example, I could send the output feature map vectors as timesteps to LSTM and it would supposedly learn dependencies between clusters of the image given the image clusters have a temporal dependency.

enter image description here

$\endgroup$
1
$\begingroup$

If you use a single layer CNN, then each vector in the resulting activation maps would be related to the original 3x3 block. However, if you stack multiple CNN layers, you increase the receptive field of each resulting vector, as shown in the image below (taken from here):

enter image description here

After the CNNs, you can certainly compute an LSTM. There are, however, some design decisions you would need to take: disregarding the batch dimension, the LSTM takes as input a sequence of vectors (2D tensor) but you have a 3D tensor (height $\times$ width $\times$ num.channels) so, how are you going to make the 3D tensor fit as input to the LSTM? You could compute the LSTM of columns/rows of the image independently, in forward or reverse direction. You could also "collapse" one dimension (e.g. by averaging or taking the maximum value).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.