Be careful with Keras Batch Normalization. You can try this code:
K.set_learning_phase(0)
input_tensor = Input(shape(img_size, img_size, 3))
base_model = ResNet50(input_tensor=input_tensor, include_top=False, weights="imagenet", pooling="avg")
x = base_model.output
#Define your own top layers
K.set_learning_phase(1)
x = Dense()
...
x = Dense()
model = Model(input_tensor, x)
for layer in base_model.layers:
layer.trainable = False
Or you can try to unfreeze the last few convolution layers, that might help. But still, be careful with Batch Normalization. There are many discussions available about this problem with keras's transfer learning.
