How to input different sized images into transfer learning network

Question

I have been looking online for a solution but have a difficult time finding a clear enough solution. I want to know how to use transfer learning (VGG16 for example) on images that have different sizes than the images the network originally trained on (so instead of inputting images of size (224,224,3) I want to input images of size (32,32,3)).

I initially thought about just padding those images but the network may look into the black pixels and think that they mean something, and I realize that might hard the accuracy and also when I tried to do that my colab notebook collapsed.

This is my VGG-16 code: def vgg16_model(img_rows, img_cols, channel=1, num_classes=None):

model = VGG16(weights='imagenet', include_top=True)

model.layers.pop()

model.outputs = [model.layers[-1].output]

model.layers[-1].outbound_nodes = []

x=Dense(num_classes, activation='relu')(model.output)

model=Model(model.input,x)

#To set the first 8 layers to non-trainable (weights will not be updated)

for layer in model.layers[:15]:

   layer.trainable = False
for layer in model.layers[16:]:
  layer.trainable=True
model_new = Sequential()
for layer in model.layers[:-1]: # just exclude last layer from copying
    model_new.add(layer)
model=model_new
model.add(Dense(256,activation='relu',input_shape=(1000,)))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(10,activation='softmax'))
#model.add(Dense(1,activation='softmax'))


# Learning rate is changed to 0.001
#sgd = SGD(lr=1e-2, decay=1e-6, momentum=0.9, nesterov=True)
sgd = SGD(lr=lr,decay=decay,momentum=0.95, nesterov=True)
adam=Adam(lr=1e-5, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0001, amsgrad=True)
#model.compile(optimizer=adam, loss='binary_crossentropy',metrics=['accuracy'])
model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['accuracy'])

# checkpoint
filepath="weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

return model

I would greatly appreciate if someone could write the corrected version of this code in order to enable me to insert pictures of size (32,32,3).

Thanks a lot in advance!!

Answer 1

Resizing is the best option, if they are bigger downscale them, else upscale them.

Answer 2

Most Imagenet pretrained CNNs were trained on 224x224 image resolution. It is a common misconception, that when using these pretrained CNN, images need to be resized to 224x224. On the contrary, popular CNN are fully convolutional nets that can accept any input size.

You can input any image size and these CNN output feature maps that are 32x times smaller. For example, if you input 224x224 then the CNN outputs feature maps of size 7x7. If you input images of size 512x512, then these CNN outputs feature maps of size 16x16.

Feature Maps
The only relevance of the pretraining 224x224 size is that these CNN have learned to find certain patterns of certain sizes. For example, maybe they learned to find circles that are 50 pixels diameter, or maybe they learned to find triangles with side length 30 pixels.

Pretrained Imagenet CNN
In the example below, we pretrain CNN on images of size 224x224 and they learn to detect circles of diameter 50 pixels and triangles of side length 30.

Why Resize Input Images
When you resize input images, you change the size of your circles and triangles. So depending on how you resize your input image, this pretrained CNN may or may not find circles of diameter 50 and triangles of side 30. In the example below, given the original image, the CNN only finds the circles when the input image is resized to 512x512 and finds triangles when resized to 128x128

So that if your input size 32x32 then your model learns from these images. And if you resize your images then your model learn those resized images.

Ref : https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147