0
$\begingroup$

I have a simple Sequential keras model with 150 Inputs. Some of these are simply OneHotEncoded values. Now I would like to add more options to the OneHotEncoder. As an example: I previously had Blue, Green and Red encoded as binary values for the input and now I want to add Yellow, Orange and Purple as well. The thing is, I would like to preserve the weights in my existing trained model and simply add new inputs with new random weights and continue training on the already established base.

How can I do that and ensure the knowledge that my model has already obtained is preserved?

If relevant: My model is saved in the .h5 format.

$\endgroup$

1 Answer 1

0
$\begingroup$

One way to do this is to create a second model with your new inputs and the same number/size of hidden layers and output layer, then copy the weights from the layers of your first model to the second model using the get_weights() and set_weights() methods of each layer. This is straightforward for all but the first hidden layer, as these layers will have the same number of weights, but a little more complex for the first layer as you need to take the new inputs into account. Here's some sample code using a couple of (untrained) toy models:

import tensorflow.keras as keras

# model1 is the pre-existing model
input_ = keras.Input((150), name='input')
x = keras.layers.Dense(8, name='dense1', activation='relu')(input_)
x = keras.layers.Dense(8, name='dense2', activation='relu')(x)
x = keras.layers.Dense(1, name='final', activation='sigmoid')(x)
model1 = keras.Model(inputs=input_, outputs=x, name='model1')

# Create model2 as new model with additional inputs
input_ = keras.Input((160), name='input')
x = keras.layers.Dense(8, name='dense1', activation='relu')(input_)
x = keras.layers.Dense(8, name='dense2', activation='relu')(x)
x = keras.layers.Dense(1, name='final', activation='sigmoid')(x)
model2 = keras.Model(inputs=input_, outputs=x, name='model2')

For model2, I've assumed the new inputs are the last ten.

I've used the default weight initialization method. You could initialize the model using the method you want to use for the new inputs, so you don't need to worry about these later.

# Check number of weights in each layer
print('Model 1 layer sizes')
for l in model1.layers:
    print(l.name, [ll.shape for ll in l.get_weights()])
print('\nModel 2 layer sizes')
for l in model2.layers:
    print(l.name, [ll.shape for ll in l.get_weights()])

This shows the models have the following numbers of weights in each layer

Model 1 layer sizes
input []
dense1 [(150, 8), (8,)]
dense2 [(8, 8), (8,)]
final [(8, 1), (1,)]

Model 2 layer sizes
input []
dense1 [(160, 8), (8,)]
dense2 [(8, 8), (8,)]
final [(8, 1), (1,)]

So each layer (apart from the input layer) has two sets of weights, the first set are the weights for the layer inputs and the second set are the biases. Get_weights returns these as numpy arrays, so we can update these as required using numpy, then update the model weights.

Now the code to update the weights in model 2.

# Copy weights for all layers apart from the input and first hidden layer
for l in range(len(model2.layers)):
    if l >= 2:
        model2.layers[l].set_weights(model1.layers[l].get_weights())

# Get the weights for the first hidden layer
l1 = model1.layers[1].get_weights()
l2 = model2.layers[1].get_weights()
# Copy biases
l2[1] = l1[1]
# Copy weights for existing inputs, assume new inputs are last.
l2[0][:len(l1[0])] = l1[0]
# Set the weights for the first hidden layer
model2.layers[1].set_weights(l2)

Print the layer weights to check

# Check results
for l in model1.layers:
    print(l.name, l.get_weights())
input []
dense1 [array([[-0.01548935, -0.02027901,  0.1186433 , ...,  0.01116119,
        -0.01081911, -0.02764229],
       ...,
       [-0.02861831, -0.09723745, -0.0620534 , ..., -0.12824798,
         0.02253188, -0.10220653]], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]
dense2 [array([[-0.01134968,  0.39225703,  0.11167014, -0.13503206,  0.09921449,
         0.27120864, -0.41560578,  0.5887881 ],
       ...,
       [ 0.23259199,  0.45360595, -0.5073748 , -0.05056351,  0.26967663,
         0.02501452, -0.03674203,  0.07765925]], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]
final [array([[-0.4500377 ],
       [ 0.7408364 ],
       [-0.73072064],
       [ 0.17347234],
       [ 0.80670106],
       [ 0.26636422],
       [ 0.5733515 ],
       [ 0.20663929]], dtype=float32), array([0.], dtype=float32)]
for l in model2.layers:
    print(l.name, l.get_weights())
input []
dense1 [array([[-0.01548935, -0.02027901,  0.1186433 , ...,  0.01116119,
        -0.01081911, -0.02764229],
       ...,
       [-0.04421869, -0.13911864,  0.09040551, ..., -0.14710264,
        -0.03600252, -0.12658373]], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]
dense2 [array([[-0.01134968,  0.39225703,  0.11167014, -0.13503206,  0.09921449,
         0.27120864, -0.41560578,  0.5887881 ],
       ...,
       [ 0.23259199,  0.45360595, -0.5073748 , -0.05056351,  0.26967663,
         0.02501452, -0.03674203,  0.07765925]], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]
final [array([[-0.4500377 ],
       [ 0.7408364 ],
       [-0.73072064],
       [ 0.17347234],
       [ 0.80670106],
       [ 0.26636422],
       [ 0.5733515 ],
       [ 0.20663929]], dtype=float32), array([0.], dtype=float32)]
$\endgroup$
1
  • $\begingroup$ Incredibly well formulated! I accepted your answer :) $\endgroup$
    – FLOROID
    Feb 23, 2023 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.