Group neural networks outputs using Keras/Tensorflow

Question

I am trying to group the outputs of my neural network, in order to have them perform a separated classification.

Let's take the example where the groups are constituted of two nodes and we previously have four output nodes, then the network should look like this :

How can I achieve this using Keras or Tensorflow ? I was thinking that I need to implement a custom layer and do the operation inside of it, but I wonder if there is an easier solution using the Keras Functional API (or lower level Tensorflow ?) :)

Answer 1

Using the keras functional API will get you what you need.

I'm assuming that you are currently using the standard keras sequential model API, which is simpler but also restricts you to a single pipeline. When using the functional API, you do need to keep track of inputs and outputs, instead of just defining layers.

For the example in your question:

from keras.layers import Input, Dense
from keras.models import Model

# Left side sub-model:
L1 = Input(shape=(2,))
L2 = Dense(2, activation='softmax')(L1)

# Right side sub-model:
R1 = Input(shape=(2,))
R2 = Dense(2, activation='softmax')(R1)

# Combining them together:
merge = concatenate([L2, R2])

# Some additional layers working on the combined layer:
merged_layer_1 = Dense(4, activation='relu')(merge)

# Output Layer:
output = Dense(2, activation='softmax')(hidden1)

# Defining the model:
my_model = Model(inputs=[L1, R1], outputs=output)

The depths of the layers are made up, but you should get the general idea.

Answer 2

Here's another Keras code snippet similar to Mark.F's answer but with the split in the reverse direction (starting with an initial input and splitting into two output branches, each with their own softmax).

from keras.layers import Dense, Input, Softmax, Concatenate, concatenate
from keras.models import Model, Sequential
import numpy as np

input_layer = Input(shape=(3,))
middle_layer = Dense(4, activation="sigmoid")(input_layer)

split_one = Dense(2, activation="sigmoid")(middle_layer)
output_one = Softmax()(split_one)

split_two = Dense(2, activation="sigmoid")(middle_layer)
output_two = Softmax()(split_two)

merged = concatenate([output_one, output_two])
model = Model(inputs=input_layer, outputs=merged)

The first two elements of the output are the result of the first branch and the second two are the result of the second branch. You'd also need to concatenate the groundtruth labels of the two branches while training.