Keras — Transfer learning — changing Input tensor shape

Question

This post seems to indicate that what I want to accomplish is not possible. However, I'm not convinced of this -- given what I've already done, I don't see why what I want to do can not be achieved...

I have two image datasets where one has images of shape (480, 720, 3) while the other has images of shape (540, 960, 3).

I initialized a model using the following code:

input = Input(shape=(480, 720, 3), name='image_input')

initial_model = VGG16(weights='imagenet', include_top=False)

for layer in initial_model.layers:
    layer.trainable = False

x = Flatten()(initial_model(input))
x = Dense(1000, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(1000, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(14, activation='linear')(x)

model = Model(inputs=input, outputs=x)
model.compile(loss='mse', optimizer='adam', metrics=['mae'])

Now that I've trained this model on the former dataset, I'd like to pop the input tensor layer off and prepend the model with a new input tensor with a shape that matches the image dimensions of the latter dataset.

model = load_model('path/to/my/trained/model.h5')
old_input = model.pop(0)
new_input = Input(shape=(540, 960, 3), name='image_input')
x = model(new_input)
m = Model(inputs=new_input, outputs=x)
m.save('transfer_model.h5')

which yields this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2506, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/models.py", line 106, in save_model
    'config': model.get_config()
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2322, in get_config
    layer_config = layer.get_config()
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2370, in get_config
    new_node_index = node_conversion_map[node_key]
KeyError: u'image_input_ib-0'

In the post that I linked, maz states that there is a dimension mismatch that prevents changing the input layer of a model -- if this was the case, then how is that I put a (480, 720, 3) input layer in front of the VGG16 model which expects (224, 224, 3) images?

I think a more likely issue is that my former model's output is expecting something different than what I'm giving it based on what fchollet is saying in this post. I'm syntactically confused, but I believe the whole x = Layer()(x) segment is constructing the layer piece by piece from input->output and simply throwing a different input in front is breaking it.

I really have no idea though...

Can somebody please enlighten me how to accomplish what I'm trying to do or, if it's not possible, explain to me why not?

Answer 1

You can do this by creating a new VGG16 model instance with the new input shape new_shape and copying over all the layer weights. The code is roughly

new_model = VGG16(weights=None, input_shape=new_shape, include_top=False)
for new_layer, layer in zip(new_model.layers[1:], model.layers[1:]):
    new_layer.set_weights(layer.get_weights())

Answer 2

The output width and height of the output dimensions of the VGGnet are a fixed portion of the input width and height because the only layers that change those dimensions are the pooling layers. The number of channels in the output is fixed to the number of filters in the last convolutional layer. The flatten layer will reshape this to get one dimension with the shape:

((input_width * x) * (input_height * x) * channels)

where x is some decimal < 1.

The main point is that the shape of the input to the Dense layers is dependent on width and height of the input to the entire model. The shape input to the dense layer cannot change as this would mean adding or removing nodes from the neural network.

One way to avoid this is to use a global pooling layer rather than a flatten layer (usually GlobalAveragePooling2D) this will find the average per channel causing the shape of the input to the Dense layers to just be (channels,) which is not dependant on the input shape to the whole model.

Once this is done none of layers in the network are dependent on the width and height of the input so the input layer can be changed with something like

input_layer = InputLayer(input_shape=(480, 720, 3), name="input_1")
model.layers[0] = input_layer

Answer 3

Here is another solution, not specific to the VGG model.

Note, that the weights of the dense layer cannot be copied (and will thus be newly initialized). This makes sense, because the shape of the weights differs in the old and the new model.

import keras
import numpy as np

def get_model():
    old_input_shape = (20, 20, 3)
    model = keras.models.Sequential()
    model.add(keras.layers.Conv2D(9, (3, 3), padding="same", input_shape=old_input_shape))
    model.add(keras.layers.MaxPooling2D((2, 2)))
    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(1, activation="sigmoid"))
    model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(lr=0.0001), metrics=['acc'], )
    model.summary()
    return model

def change_model(model, new_input_shape=(None, 40, 40, 3)):
    # replace input shape of first layer
    model._layers[1].batch_input_shape = new_input_shape

    # feel free to modify additional parameters of other layers, for example...
    model._layers[2].pool_size = (8, 8)
    model._layers[2].strides = (8, 8)

    # rebuild model architecture by exporting and importing via json
    new_model = keras.models.model_from_json(model.to_json())
    new_model.summary()

    # copy weights from old model to new one
    for layer in new_model.layers:
        try:
            layer.set_weights(model.get_layer(name=layer.name).get_weights())
        except:
            print("Could not transfer weights for layer {}".format(layer.name))

    # test new model on a random input image
    X = np.random.rand(10, 40, 40, 3)
    y_pred = new_model.predict(X)
    print(y_pred)

    return new_model

if __name__ == '__main__':
    model = get_model()
    new_model = change_model(model)

Answer 4

This should be pretty easy with kerassurgeon. First you need to install the library; depending on if you are using Keras through TensorFlow (with tf 2.0 and up) or Keras as a separate library, it needs to be installed in different ways.

For Keras in TF: pip install tfkerassurgeon (https://github.com/Raukk/tf-keras-surgeon). For standalone Keras: pip install kerassurgeon (https://github.com/BenWhetton/keras-surgeon)

To replace the input (example with TF 2.0; currently untested code):

from tensorflow import keras  # or import keras for standalone version
from tensorflow.keras.layers import Input

model = load_model('path/to/my/trained/model.h5')
new_input = Input(shape=(540, 960, 3), name='image_input')

# or kerassurgeon for standalone Keras
from tfkerassurgeon import delete_layer, insert_layer

model = delete_layer(model.layers[0])
# inserts before layer 0
model = insert_layer(model.layers[0], new_input)

Answer 5

@gebbissimo answer worked for me in TF2 with just small adaptations that I share below in a single function:

def change_input_size(model,h,w,ch=3):
   model._layers[0]._batch_input_shape = (None,h,w,ch)
   new_model = keras.models.model_from_json(model.to_json())
   new_model.summary()
   for layer,new_layer in zip(model.layers,new_model.layers):
      new_layer.set_weights(layer.get_weights())
   return new_model

Answer 6

This how I change the input size in Keras model. I have two CNN models, one with input size [None, None, 3] while the other has input size [512,512,3]. Both models have the same weights. By using set_weights(model.get_weights()), weights of model 1 can be transferred to model 2

inputs = Input((None, None, 3))
.....
model = Model(inputs=[inputs], outputs=[outputs])
model.compile(optimizer='adam', loss='mean_squared_error')
model.load_weights('my_model_name.h5')

inputs2 = Input((512, 512, 3))
....
model2 = Model(inputs=[inputs2], outputs=[outputs])
model2.compile(optimizer='adam', loss='mean_squared_error')
model2.set_weights(model.get_weights())