# What is the significance of model merging in Keras?

I have learned that Keras has a functionality to "merge" two models according to the following:

from keras.layers import Merge

left_branch = Sequential()

right_branch = Sequential()

merged = Merge([left_branch, right_branch], mode='concat')


What is the point in mergint NNs, in which situations is it useful? Is it a kind of ensemble modelling? What is the difference between the several "modes" (concat, avg, dot etc...) in the sense of performance?

It is used for several reasons, basically it's used to join multiple networks together. A good example would be where you have two types of input, for example tags and an image. You could build a network that for example has:

IMAGE -> Conv -> Max Pooling -> Conv -> Max Pooling -> Dense

TAG -> Embedding -> Dense layer

To combine these networks into one prediction and train them together you could merge these Dense layers before the final classification.

Networks where you have multiple inputs are the most 'obvious' use of them, here is a picture that combines words with images inside a RNN, the Multimodal part is where the two inputs are merged:

Another example is Google's Inception layer where you have different convolutions that are added back together before getting to the next layer.

To feed multiple inputs to Keras you can pass a list of arrays. In the word/image example you would have two lists:

x_input_image = [image1, image2, image3]
x_input_word = ['Feline', 'Dog', 'TV']
y_output = [1, 0, 0]


Then you can fit as follows:

model.fit(x=[x_input_image, x_input_word], y=y_output]

• Sorry, I cannot see the point in building separate networks for both the training instances and the labels while there is a possibility to feed these in a single network in the fitting phase which does the job anyway. I can see that merging is a possibility but not its advantage over "non-merging". – Hendrik Aug 16 '16 at 7:59
• How do you feed them in the fitting phase? The inputs are always seperate, you cannot use your convolution layer on your labels so these layers need to be merged somehow. – Jan van der Vegt Aug 16 '16 at 8:00
• In Keras model.fit() accepts both X and y for fitting and model in this case can be an "non-merged" model as well. Pretty much like other model types in Sklearn for example. – Hendrik Aug 16 '16 at 8:13
• Labels might be a poorly chosen name from my side, let's say you have a picture and the annotation with that picture, and you want to classify if that combination is about cats or not, then you have two types of input, and one binary output. To get the synergy between them you will have to merge the layers somewhere. Another example is where you have two pictures, one from the top and one from the bottom that you have to classify together – Jan van der Vegt Aug 16 '16 at 8:16
• @Hendrik: There aren't "component models", there is only one model. It is a complex one, enabled by the layer merging feature. You evaluate it as you do for any single model - i.e. with a metric against a hold-out test data set (in the image/words example with data comprising images, associated partial text and the next word as the label to predict). If you want, you can inspect the layers within the model to see what they are doing - e.g. the analysis of CNN features can still be applied to the convolutional layers. – Neil Slater Aug 16 '16 at 8:52