What is the significance of model merging in Keras?

Question

I have learned that Keras has a functionality to "merge" two models according to the following:

from keras.layers import Merge

left_branch = Sequential()
left_branch.add(Dense(32, input_dim=784))

right_branch = Sequential()
right_branch.add(Dense(32, input_dim=784))

merged = Merge([left_branch, right_branch], mode='concat')

What is the point in mergint NNs, in which situations is it useful? Is it a kind of ensemble modelling? What is the difference between the several "modes" (concat, avg, dot etc...) in the sense of performance?

Answer 1

It is used for several reasons, basically it's used to join multiple networks together. A good example would be where you have two types of input, for example tags and an image. You could build a network that for example has:

IMAGE -> Conv -> Max Pooling -> Conv -> Max Pooling -> Dense

TAG -> Embedding -> Dense layer

To combine these networks into one prediction and train them together you could merge these Dense layers before the final classification.

Networks where you have multiple inputs are the most 'obvious' use of them, here is a picture that combines words with images inside a RNN, the Multimodal part is where the two inputs are merged:

Another example is Google's Inception layer where you have different convolutions that are added back together before getting to the next layer.

To feed multiple inputs to Keras you can pass a list of arrays. In the word/image example you would have two lists:

x_input_image = [image1, image2, image3]
x_input_word = ['Feline', 'Dog', 'TV']
y_output = [1, 0, 0]

Then you can fit as follows:

model.fit(x=[x_input_image, x_input_word], y=y_output]