I also asked this question on Stack Overflow. However, it has not yet been answered and I think this is a more suitable platform to place it.

I'm trying to implement a network set-up similar to this Google Deepmind paper. Their network set-up is as follows:

enter image description here

Mθ is a convolutional network, so I was wondering how they concatenate the input frames with the viewpoints? As far as I knew, the CNN takes into account the spatial information right? So does it even make sense to concatenate the frames with the viewpoints as an input for the CNN?

Thanks in advance!


I was thinking about it and they might have concatenated the viewpoint on the dense layers behind the convolution layers. Might that be the case?


1 Answer 1


Here the viewpoint ($v_i$) and corresponding frame ($f_i$) are not concatenated. $v_i$ is simply an index of $f_i$. As mentioned in their section 3.1, $v_i$ is the timestamp and $f_i$ is the actual frame (image). The convolution network $M_\theta$ is applied on $f_i$, not $v_i$.

After the convolution network $M_\theta$, the output of different frames $r_i$ are added together (notice the "+" in the figure, as well as the equation $r=r_1+...+r_m$ in section 3.2.

  • $\begingroup$ You’re right 🤦🏻‍♂️. I was thinking in my problem so thats why I was confused. I’m doing the same, but I have a timestamp instead of a viewpoint. $\endgroup$ Commented Sep 11, 2018 at 15:46
  • $\begingroup$ Could you have a look at section 4.1 of the paper under "Training CGQN"? Here they say: the model is given between m1 and m2 frames (and corresponding timestamps). So did they input the timestamps here? $\endgroup$ Commented Sep 12, 2018 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.