# Combining spatial input with a label as input for CNN using Keras

I also asked this question on Stack Overflow. However, it has not yet been answered and I think this is a more suitable platform to place it.

I'm trying to implement a network set-up similar to this Google Deepmind paper. Their network set-up is as follows:

Mθ is a convolutional network, so I was wondering how they concatenate the input frames with the viewpoints? As far as I knew, the CNN takes into account the spatial information right? So does it even make sense to concatenate the frames with the viewpoints as an input for the CNN?

Here the viewpoint ($v_i$) and corresponding frame ($f_i$) are not concatenated. $v_i$ is simply an index of $f_i$. As mentioned in their section 3.1, $v_i$ is the timestamp and $f_i$ is the actual frame (image). The convolution network $M_\theta$ is applied on $f_i$, not $v_i$.
After the convolution network $M_\theta$, the output of different frames $r_i$ are added together (notice the "+" in the figure, as well as the equation $r=r_1+...+r_m$ in section 3.2.