look careful of BN.You can try this code:

K.set_learning_phase(0)
input_tensor = Input(shape(img_size, img_size, 3))  
base_model = ResNet50(input_tensor=input_tensor, include_top=False, weights="imagenet", pooling="avg")
x = base_model.output
#Define your own top layers
K.set_learning_phase(1)
x = Dense()
...
x = Dense()
model = Model(input_tensor, x)
for layer  in base_model.layers:
    layer.trainable = False

Or you can try to unfreeze last few convolution layers, mabey help.But still, watch out BN. There are many talk about this problem of keras's transfer learning.

Be careful with Keras Batch Normalization. You can try this code:

K.set_learning_phase(0)
input_tensor = Input(shape(img_size, img_size, 3))  
base_model = ResNet50(input_tensor=input_tensor, include_top=False, weights="imagenet", pooling="avg")
x = base_model.output
#Define your own top layers
K.set_learning_phase(1)
x = Dense()
...
x = Dense()
model = Model(input_tensor, x)
for layer  in base_model.layers:
    layer.trainable = False

Or you can try to unfreeze the last few convolution layers, that might help. But still, be careful with Batch Normalization. There are many discussions available about this problem with keras's transfer learning.