What is exactly meant by neural network that can take different types of input?

Question

There is a scientific document that implements a convolutional neural network to classify 3 different types of data, although how exactly, is unknown to me.

Here's the explanation of network architecture:

This section describes architecture of our neural net which is depicted in Fig. 3. Our network has three types of inputs: Screenshot (we use upper crop of the page with dimensions 1280 × 1280, however this net can work with arbitrarily sized pages), TextMaps (tensor with dimensions $128 \times 160 \times 160$) and Candidate boxes (list of box coordinates of arbitrary length).

A screenshot is processed by three convolutional layers (the first two layers are initialized with pretrained weights from BVLC AlexNet). TextMaps are processed with one convolutional layer with kernel size $1 \times 1$ and thus its features capture various combinations of words. These two layers are then concatenated and processed by final convolutional layer.

---

What exactly is implied by Our network has three types of inputs above? Is it possible for convolutional neural network to pass different types of inputs differently?

From my understanding, neural network for Screenshot input would be created like this:

def CNN(features, labels, mode):
  input_layer = tf.reshape(image, [-1, 1280, 1280, 1])

  # Conv+ReLU
  conv_relu_1 = tf.layers.conv2d(
      inputs=input_layer,
      filters=96,
      kernel_size=[11, 11],
      padding="same",
      activation=tf.nn.relu)

  # MaxPool
  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[3, 3], strides=2)

  # Conv + ReLU
  ...

So let's say this is first neural network, then should I create another neural network for TextMaps and concatenate results? Or does every magic just happen in a single neural network?

In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?

Thank you!

Answer 1

In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?

Yes, you can. Check the Functional API of Keras, on how to define multi input/output networks. Then you can create different models for the processing of each input and fuse them together into a single multi-input model using the keras.models.Model() class.

In the following example, you can see that the main_input is processed differently than the aux_input and both are thereafter merged together to be propagated through the rest of the layers of the network.