Is there a method to apply trained weights to a model with the same input shape and model architecture but different output shape?

Question

I am developing a speaker identification model in Keras, and I have saved the weights from a trained custom model. Now, I am looking to use the trained weights to fine tune the model on a new dataset, but I am having trouble since the new dataset contains a different number of speakers than the first, so the new output shape will be different from the original.

Here's the code that I am using to create and evaluate the model:

# Create Model
def createModel(model_input, model_output, first_session=True):
    
    # Define Input Layer
    inputs = model_input
    
    # Define First Conv2D Layer
    conv = Conv2D(filters=32,
                  kernel_size=(5, 5),
                  activation='relu',
                  padding='same',
                  strides=3)(inputs)
    conv = Conv2D(filters=32,
                  kernel_size=(5, 5),
                  activation='relu',
                  padding='same',
                  strides=3)(conv)
    conv = MaxPooling2D(pool_size=(3, 3), padding='same')(conv)
    conv = Dropout(0.3)(conv)
    
    # Define Second Conv2D Layer
    conv = Conv2D(filters=64,
                  kernel_size=(3, 3),
                  activation='relu',
                  padding='same',
                  strides=3)(conv)
    conv = Conv2D(filters=64,
                  kernel_size=(3, 3),
                  activation='relu',
                  padding='same',
                  strides=3)(conv)
    conv = MaxPooling2D(pool_size=(3, 3), padding='same')(conv)
    conv = Dropout(0.3)(conv)
    
    # Define Third Conv2D Layer
    conv = Conv2D(filters=128,
                  kernel_size=(3, 3),
                  activation='relu',
                  padding='same',
                  strides=3)(conv)
    conv = Conv2D(filters=128,
                  kernel_size=(3, 3),
                  activation='relu',
                  padding='same',
                  strides=3)(conv)
    conv = MaxPooling2D(pool_size=(3, 3), padding='same')(conv)
    conv = Dropout(0.3)(conv)
    
    # Define Flatten Layer
    conv = Flatten()(conv)
    
    # Define First Dense Layer
    conv = Dense(256, activation='relu')(conv)
    conv = Dropout(0.2)(conv)
    
    # Define Second Dense Layer
    conv = Dense(128, activation='relu')(conv)
    conv = Dropout(0.2)(conv)
    
    # Define Output Layer
    outputs = Dense(model_output, activation='softmax')(conv)
    
    # Create Model
    model = Model(inputs, outputs)
    
    model.summary()
    
    if first_session != True:
        model.load_weights('SI_ideal_weights_simple.hdf5')
    
    return model

# Train Model
def evaluateModel(x_train, x_val, y_train, y_val, num_classes, first_session=True):
    
    # Model Parameters
    verbose, epochs, batch_size, patience = 1, 100, 64, 10
    
    # Determine Input and Output Dimensions
    x = x_train[0].shape[0] # Number of MFCC rows
    y = x_train[0].shape[1] # Number of MFCC columns
    c = 1 # Number of channels
    n_outputs = num_classes # Number of outputs
    
    # Create Model
    inputs = Input(shape=(x, y, c))
    
    model = createModel(model_input=inputs, 
                         model_output=n_outputs,
                         first_session=first_session)
    
    # Compile Model
    model.compile(loss='categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

    # Callbacks
    es = EarlyStopping(monitor='val_loss',
                       mode='min',
                       verbose=verbose,
                       patience=patience,
                       min_delta=0.0001) # Stop training at right time
    
    mc = ModelCheckpoint('SI_ideal_weights_simple.hdf5',
                         monitor='val_accuracy',
                         verbose=verbose,
                         save_weights_only=True,
                         save_best_only=True,
                         mode='max') # Save best model after each epoch
    
    reduce_lr = ReduceLROnPlateau(monitor='val_loss',
                                  factor=0.2,
                                  patience=patience//2,
                                  min_lr=1e-3) # Reduce learning rate once learning stagnates
    
    # Evaluate Model
    model.fit(x=x_train, y=y_train, epochs=epochs,
              callbacks=[es,mc,reduce_lr], batch_size=batch_size,
              validation_data=(x_val, y_val))
    
    accuracy = model.evaluate(x=x_train, y=y_train, 
                              batch_size=batch_size,
                              verbose=verbose)
    
    return (accuracy[1], model)

Attempting to run the model on the second dataset throws the following error:

ValueError: Shapes (128, 40) and (128, 15) are incompatible

Which occurs at the output layer because of the difference in the number of speakers (i.e. from 40 to 15). The last layer contains 5160 trainable parameters, so I was trying to find a solution other than dropping it and adding an equivalent one with a new output shape to retain accuracy, if possible. (That being said, I am new to ML/Keras, and I can't say for certain that this would make a substantial difference.)

Ultimately, my question is: How can I apply the weights from a custom trained convolutional neural net to a dataset with the same data shape but different number of classes?

Any help is greatly appreciated.

Answer 1

Artificial Neural Networks are like a black box learning system. There is no known, or generally agreed upon, method that dictates what each weight represents or means for a given learning problem. Its internal representation of the problem is opaque to the architect.

In fact, the final trained weights are very closely tied to the neural network architecture and is very logical to assume that they cannot be transfered to another arbitrary architecture of another neural network.

That being said, there is research related to re-purposing an already trained neural network to another similar task. This is called Transfer Learning in machine learning literature.

Some resources to get you started:

A Gentle Introduction to Transfer Learning for Deep Learning

Transfer learning only works in deep learning if the model features learned from the first task are general.

How transferable are features in deep neural networks?

Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.

Deep Learning using Transfer Learning

• What to transfer — We need to understand what knowledge is common between the source and target task. What knowledge can be transferred from source task to target task that will help improve the performance of the target task

• When to transfer or when not to Transfer - When the source and target domains are not related at all we should not try to apply transfer learning. In such a scenario the performance will suffer. This type of transfer is called Negative Transfer. We should apply Transfer learning only when source and target domains/tasks are related

• How to transfer: Identifying different techniques to apply transfer learning when the source and target domain/task are related. We can use Inductive transfer learning, Transductive transfer learning or unsupervised transfer learning.

An overview of attempts to interpret deep-learning models and a new suggestion in Causality Learning: A New Perspective for Interpretable Machine Learning

Recent years have witnessed the rapid growth of machine learning in a wide range of fields such as image recognition, text classification, credit scoring prediction, recommendation system, etc. In spite of their great performance in different sectors, researchers still concern about the mechanism under any machine learning (ML) techniques that are inherently black-box and becoming more complex to achieve higher accuracy. Therefore, interpreting machine learning model is currently a mainstream topic in the research community. However, the traditional interpretable machine learning focuses on the association instead of the causality. This paper provides an overview of causal analysis with the fundamental background and key concepts, and then summarizes most recent causal approaches for interpretable machine learning. The evaluation techniques for assessing method quality, and open problems in causal interpretability are also discussed in this paper.

Now, to answer your main question given the already mentioned points, is to try heuristics in a trial and error manner, there is no standard procedure.

For example you can set superfluous output weights to zero, or missing output weights to zero. One can try other linear (or non-linear) combinations to change the amount of output weights to match the original neural network to the new dataset.

One can even train a neural net whose sole purpose is to adapt the output classes of the original network to the output classes of the new problem, and concatenate it with the orignal neural net. However if one takes this approach, why not train a new convolutional network from scratch that directly classifies the new problem.

Answer 2

This should work -

- Make your last layer as second last layer with activation='relu'
- Assign weights from the previous model
- Add a layer on top of it with 15 Neurons and activation='softmax'.

Answer 3

I worked on something similar where my model categorizes a title. My initial model had a fixed output size of 40 categories. The number of categories was supposed to be up-scalable as necessary.

model_main = Sequential()
model_main.add(Embedding(input_dim=30522, output_dim=100, input_length=100))
model_main.add(BatchNormalization())
model_main.add(Bidirectional(LSTM(128, dropout=0.2, recurrent_dropout=0.2, return_sequences=True)))
model_main.add(BatchNormalization())
model_main.add(Bidirectional(LSTM(128, dropout=0.2, recurrent_dropout=0.2)))
model_main.add(BatchNormalization())
model_main.add(Dense(40, activation = 'softmax'))
model_main.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=[Precision()])

Now, I trained and saved this model along with its history.

hisory_main = model_main.fit(X_train, y_main_train, epochs=30, batch_size=16)

pickle.dump(hisory_main.history, open(('history_main.pkl'), 'wb'))
model_main.save('model_main.keras')

Now, I want to add 5 more categories. I did this by creating a new model and transferred each layer except for the output layer from the stored model. As of for the output dense layer, I copied the weights and biases of the stored model for the first 40 neurons and initialized the new 5 neurons with zero weights. If you don't want the previous model, you can directly modify the last layer of the previous model directly without creating a new model.

model_main = load_model('model_main.keras')

new_model_main = Sequential()
for layer in model_main.layers[:-1]:
    new_model_main.add(layer)  

weights, biases = model_main.layers[-1].get_weights()

new_weights = np.zeros((weights.shape[0], 45))
new_weights[:, :weights.shape[1]] = weights  

new_biases = np.zeros((45,))
new_biases[:biases.shape[0]] = biases

new_model_main.add(Dense(45, activation='softmax'))
new_model_main.layers[-1].set_weights([new_weights, new_biases])

new_model_main.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=[Precision()])

For the training of this, since I wanted the previous weights to be propagated, I trained the model right where I left off by using the history file.

with open('history_main.pkl', 'rb') as file:
    history_main = pickle.load(file)

num_epochs_main = len(history_main['loss'])

new_history_main = model_main.fit(X, y_main, epochs=(num_epochs_main + 2), batch_size=16, initial_epoch=num_epochs_main)

for key in history_main:
    history_main[key] += new_history_main.history[key]

In your case where you want to scale down, If you want to use the first 15 speakers weights, I believe you have to copy only the first 15 neurons from the output layer.

PS: This is my first time answering to something. Pardon me if something's not right.