Before going into an obvious XY problem, I will explain you what I'm trying to do.
I'm training a simple MobileNet pre-trained with Imagenet for multiclass classification. What I do is freeze all the convolutional part, and then create a new Prediction Layer (conv2D 1x1xN where N is the number of classes).
Let's say I trained with 5 classes, and then I got a new set of images of 5 new classes. I want to keep the knowledge from the last training and be able to train only with the new classes.
What I do is to train a new model with the same frozen pre-trained weights, but only with the new 5 classes, then concatenate the weights from the old and the new model into a last layer that outputs for 10 classes.
The concatenation works perfectly, but when I run the evaluation I got a horrible accuracy (like if it was predicted randomly). For example, for 30 clasess trained in "sets" of 5, I got an accuracy of 0.03.
Is what I'm doing viable? A problem that I think it can be causing this is that the "labels" for every prediction are not kept in the same order or something, so that even if I copy the weights, the predictions may be unordered.