Prepending Input layer to pre-trained model

Question

I'm trying to input numpy arrays of shape (1036800,) - originally images of shape (480, 720, 3) - into a pre-trained VGG16 model to predict continuous values.

I've tried several variations of the code below:

input = Input(shape=(1036800,), name='image_input')
initial_model = VGG16(weights='imagenet', include_top=False)
x = Flatten()(initial_model(input).output)
x = Dense(200, activation='relu')(x)
x = Dense(1)(x)
model = Model(inputs=input, outputs=x)

Previous variations of the above code yielded errors related to the input being the wrong dimensions, input_shape needing to have 3 channels (when using (1036800,) for that parameter in the initialization of VGG16), and the most recent error that results from running the above code is this:

Traceback (most recent call last):
  File "model_alex.py", line 57, in <module>
    model = initialize_model()
  File "model_alex.py", line 20, in initialize_model
    x = Flatten()(initial_model(input).output)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 596, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2061, in call
    output_tensors, _, _ = self.run_internal_graph(inputs, masks)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2212, in run_internal_graph
    output_tensors = _to_list(layer.call(computed_tensor, **kwargs))
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/layers/convolutional.py", line 164, in call
    dilation_rate=self.dilation_rate)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3156, in conv2d
    data_format='NHWC')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 639, in convolution
    input_channels_dim = input.get_shape()[num_spatial_dims + 1]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 500, in __getitem__
    return self._dims[key]
IndexError: list index out of range

Here is the full code. Here is the sample data file used in the script.

One of the possible approach towards fixing this might be to resize the raw image files to 224x224 and turn them into numpy arrays of shape (224, 224, 3) so they can be plugged into the pre-trained model's first layer. However, I don't want to warp the images or waste another night pre-processing data when I should already be training.

Besides that, I all I can think to do is Google my problem and try to adapt the found solutions or aimlessly tweak various shape related parameters and functions -- neither of which has gotten me very far over the past 4 hours.

Answer 1

The issue is that you shouldn't flatten the images into 1-dimensional vector because the VGG16 contains 2D convolution layers (e.g. spatial convolution over images), which require the input to have the shape of (number_of_images, image_height, image_width, image_channels), given that keras.backend.image_data_format() returns 'channels_last'. If your image_data_format is 'channels_first', change the input data shape to (number_of_images, image_channels, image_height, image_width).

Here is your fixed code (tested with Keras 2.0.4):

x_train = x_train.reshape((x_train.shape[0], 480, 720, 3))
x_test = x_test.reshape((x_test.shape[0], 480, 720, 3))

initial_model = VGG16(weights='imagenet', include_top=False)
input = Input(shape=(480, 720, 3), name='image_input')
x = Flatten()(initial_model(input))
x = Dense(200, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(1)(x)
model = Model(inputs=input, outputs=x)
model.compile(loss='mse', optimizer='adam')

model.fit(x_train, y_train, epochs=20, batch_size=16)
score = model.evaluate(x_test, y_test, batch_size=16)