1
$\begingroup$

There is a scientific document that implements a convolutional neural network to classify 3 different types of data, although how exactly, is unknown to me.

Here's the explanation of network architecture:

This section describes architecture of our neural net which is depicted in Fig. 3. Our network has three types of inputs: Screenshot (we use upper crop of the page with dimensions 1280 × 1280, however this net can work with arbitrarily sized pages), TextMaps (tensor with dimensions $128 \times 160 \times 160$) and Candidate boxes (list of box coordinates of arbitrary length).

A screenshot is processed by three convolutional layers (the first two layers are initialized with pretrained weights from BVLC AlexNet). TextMaps are processed with one convolutional layer with kernel size $1 \times 1$ and thus its features capture various combinations of words. These two layers are then concatenated and processed by final convolutional layer.

enter image description here


What exactly is implied by Our network has three types of inputs above? Is it possible for convolutional neural network to pass different types of inputs differently?

From my understanding, neural network for Screenshot input would be created like this:

def CNN(features, labels, mode):
  input_layer = tf.reshape(image, [-1, 1280, 1280, 1])

  # Conv+ReLU
  conv_relu_1 = tf.layers.conv2d(
      inputs=input_layer,
      filters=96,
      kernel_size=[11, 11],
      padding="same",
      activation=tf.nn.relu)

  # MaxPool
  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[3, 3], strides=2)

  # Conv + ReLU
  ...

So let's say this is first neural network, then should I create another neural network for TextMaps and concatenate results? Or does every magic just happen in a single neural network?

In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?

Thank you!

$\endgroup$
4
$\begingroup$

In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?

Yes, you can. Check the Functional API of Keras, on how to define multi input/output networks. Then you can create different models for the processing of each input and fuse them together into a single multi-input model using the keras.models.Model() class.

In the following example, you can see that the main_input is processed differently than the aux_input and both are thereafter merged together to be propagated through the rest of the layers of the network.

enter image description here

$\endgroup$
  • $\begingroup$ Hello, thank you for the answer, name "multi-input/output convultional neural network" seems to be a key word here. Are you aware of multi-input convolutional neural networks in raw tensorflow (or is it better to use high-level library for this purpose?). $\endgroup$ – ShellRox Jan 19 at 19:28
  • 1
    $\begingroup$ Hi, to be honest I mostly work on a higher level with Keras and I strongly believe that you will find it quite easy to implement this kind of architecture in Keras, similarly to the attached image :) $\endgroup$ – pcko1 Jan 19 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.