I'm trying to grasp the structure of this convoluted neural network.



I understand the first layer is a 6x6 conv with stride 2 followed by 3x3 max pool and then 6 5x5 convs and another 3x3 max pool. After this, however, outputs from a fully connected layer of 64 neurons are "tiled over the special dimensions of the response map of pool2".

I don't understand what this means. The output of pool2 should be 64 (because of 64 filters) 18x18 arrays. In the first 18x18 array do I add output1 to each of the 18*18=324 values, and in the second array add output2 to each of the 324 values, etc.?


What do I do with the 64 outputs (each is a 18x18 array) and the 64 outputs from the fully connected layer?


The so-called motor command $v_t$ (I don't know what it means but it looks to be some scalar feature although it would work the same if it's a vector) is fed into a layer that builds 64 representations of this value, one for each feature map in the convolutional layer that we are going to add it too. The conv layers have a spatial resolution however, and this representation is only one number for the corresponding feature map. What they do is tile this number so that this whole motor command representation has the same dimensions as the max pooling layer after the convolutional layer (pool2). Now that the dimensions match we can use an element-wise addition operation to inject this information into the convolutional network.

  • $\begingroup$ Thanks so much for your help! I thought that was likely the case but wasnt entirely sure. $\endgroup$ Oct 4 '17 at 15:16
  • $\begingroup$ Another question: The paper mentions that the two input images shown as i-0 and i-t in the image are supposed to be concatenated prior to being provided to the network. Does this mean to take the two 472x472 images and put them into one (472*2)x472 image where the images are side by side? $\endgroup$ Oct 5 '17 at 2:54
  • $\begingroup$ It's not super clear from the paper, but you would usually stack them in the depth channel, if they are the same resolution. Both your images are 472x472x3 in width x height x depth (colors). After concatenating in the depth dimension you will get to 472x472x6. It makes sense to concatenate them there, so that the same parts of the image are used in the same convolutions. $\endgroup$ Oct 5 '17 at 7:32
  • $\begingroup$ i assume that the last fully connected layer in the network is followed by a output layer of 1 neuron that spits out the probability through a sigmoid. I do know that it mentioned the sigmoid. Never said anything about the output layer... Do you think that would be correct? $\endgroup$ Oct 7 '17 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.