1
$\begingroup$

I have several models that classify the input (word embedding) into several classes. My problem is that I need to train these models separately and need to merge the output of these models together to get a label.

For simplicity assume only two models:

Model 1: predicts A, B or C
Model 2: predicts D or E

Then, I need to classify the input X so that I get the joint probability over A, B, C, D and E.

I have tried this code in Keras/Python, in which I define neural networks and remove the softmax activation function after training:

inputs = tf.keras.Input(shape=dim_in)
x = layers.Dense(dim_out)(inputs)
outputs = layers.Activation(activation="softmax")(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs, name='keras_model')
model_without_softmax = tf.keras.Model(inputs=model.inputs[0], outputs=model.layers[-2].output, name='keras_model', trainable=False)

model.compile(optimizer=tf.keras.optimizers.Nadam(),
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                  metrics=['sparse_categorical_accuracy'])

Then I get final prediction with softmax over joint arrays of predictions:

prediction = softmax(np.append(prediction_1, prediction_2))

It wasn't working because the output values from the neural networks were on a different scale when classifying into 2 or 3 classes. Thus, I tried dividing into a number of classes but without success:

prediction = softmax(np.append(prediction_1/len(idx2label_d1), prediction_2/len(idx2label_d2)))

Can you point me to right direction for the general formula of merging the neural network's outputs together?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Maybe you can use the Concatenate layer

outputs = tf.keras.Concatenate()([model1, model2])
full_model = tf.keras.Model(inputs=inputs, outputs=outputs, name='full_model')

This will simply concatenate the two softmax output into one.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The problem is that I don't have the opportunity to train it in a supervised way together - imagine that I have one system which knows only his labels and the other which knows only his labels and they cannot see the labels of each other in train phase but I need to merge their predictions in inference phase. $\endgroup$
    – United121
    Jul 9, 2020 at 7:50
  • $\begingroup$ There is no need to train them together, you can use model1 and model2 with your pretrained weights learned separately. You create model3 after training separately, and do not need to train model3, since it has no new weights (the Concatenate layer simply concatenate outputs of model1 and model2) $\endgroup$
    – Adam Oudad
    Jul 9, 2020 at 13:48
  • $\begingroup$ I get it .. so I tried the performance with softmax activation at the end - the result (f1 score) for my testing set was 0.68, but if I use only one model (which is possible in my research environment) I got f1 score 0.88 - do you think that it is not possible to get closer to that one model score? $\endgroup$
    – United121
    Jul 9, 2020 at 19:46
  • $\begingroup$ Maybe you can try with softmax on 5 output values for both model1 and model2. You train them separately by adapting their respective training data, that is adding zero values to the labels they are not trained to classify. Then you can try to add both softmax of model1 and model2 to get the output of model3. $\endgroup$
    – Adam Oudad
    Jul 10, 2020 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.