There is a scientific document that implements a convolutional neural network to classify 3 different types of data, although how exactly, is unknown to me.
Here's the explanation of network architecture:
This section describes architecture of our neural net which is depicted in Fig. 3. Our network has three types of inputs: Screenshot (we use upper crop of the page with dimensions 1280 × 1280, however this net can work with arbitrarily sized pages), TextMaps (tensor with dimensions $128 \times 160 \times 160$) and Candidate boxes (list of box coordinates of arbitrary length).
A screenshot is processed by three convolutional layers (the first two layers are initialized with pretrained weights from BVLC AlexNet). TextMaps are processed with one convolutional layer with kernel size $1 \times 1$ and thus its features capture various combinations of words. These two layers are then concatenated and processed by final convolutional layer.
What exactly is implied by Our network has three types of inputs above? Is it possible for convolutional neural network to pass different types of inputs differently?
From my understanding, neural network for Screenshot input would be created like this:
def CNN(features, labels, mode): input_layer = tf.reshape(image, [-1, 1280, 1280, 1]) # Conv+ReLU conv_relu_1 = tf.layers.conv2d( inputs=input_layer, filters=96, kernel_size=[11, 11], padding="same", activation=tf.nn.relu) # MaxPool pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[3, 3], strides=2) # Conv + ReLU ...
So let's say this is first neural network, then should I create another neural network for TextMaps and concatenate results? Or does every magic just happen in a single neural network?
In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?