Combine two separate models created via Transfer Learning?

Question

Suppose have two 'image classification' models created by transfer learning on the same base model[1], each producing a different set of labels/classes. Trained at different times, with different training images. Just Transfer learning, not fine-tuning.

As understand it when training, the initial layers are extracted from the base model, and 'frozen' so only the final layers are retrained, so in concept it seems the both final models will have common initial layers, with different output dimensions.

Is there a way to join or combine the final models, to create a single model that can predict all the labels. The input would be the same but output would be a single longer list of labels?

Say each model produced predictions for 5 labels/classes. The new combined model would produce 10 labels. The output layer would be bigger. Penultimate layers may need to grow too, although the input and initial layers would be unchanged.

Expect this is a relatively common operation, but unsure on terminology, merge/combine/concatenate? Ultimately would be doing it in python, on ready created models.

[1] EfficientNet B0 using liner.ai

Answer 1

If you know what was the final frozen layer, I imagine you can create a new model that starts with the base model up to that layer, and then runs two separate branches each with the trained weights of the different transfer-learned models. Once you get the final output predictions for the two branches, concatenate and softmax them.

That would be the hacky way to achieve what you want, and does not require further training, but it has drawbacks. For example, assume model A predicts labels 1-5 and model B predicts labels 6-10. Given a sample with true label 6, nothing guarantees that the largest probability model B outputs will be higher than the largest probability output by model A.

If you can verify that model A outputs close to uniform probabilities on labels learned by model B (and vice versa), this approach may prove useful, but otherwise you will have to retrain the hybrid model on all your training data.