Combining spatial input with a label as input for CNN using Keras

Question

I also asked this question on Stack Overflow. However, it has not yet been answered and I think this is a more suitable platform to place it.

I'm trying to implement a network set-up similar to this Google Deepmind paper. Their network set-up is as follows:

M_θ is a convolutional network, so I was wondering how they concatenate the input frames with the viewpoints? As far as I knew, the CNN takes into account the spatial information right? So does it even make sense to concatenate the frames with the viewpoints as an input for the CNN?

Thanks in advance!

Edit

I was thinking about it and they might have concatenated the viewpoint on the dense layers behind the convolution layers. Might that be the case?

Answer 1

Here the viewpoint ($v_i$) and corresponding frame ($f_i$) are not concatenated. $v_i$ is simply an index of $f_i$. As mentioned in their section 3.1, $v_i$ is the timestamp and $f_i$ is the actual frame (image). The convolution network $M_\theta$ is applied on $f_i$, not $v_i$.

After the convolution network $M_\theta$, the output of different frames $r_i$ are added together (notice the "+" in the figure, as well as the equation $r=r_1+...+r_m$ in section 3.2.