look careful of BN.You can try this code:
```
K.set_learning_phase(0)
input_tensor = Input(shape(img_size, img_size, 3))  
base_model = ResNet50(input_tensor=input_tensor, include_top=False, weights="imagenet", pooling="avg")
x = base_model.output
#Define your own top layers
K.set_learning_phase(1)
x = Dense()
...
x = Dense()
model = Model(input_tensor, x)
for layer  in base_model.layers:
    layer.trainable = False
```

Or you can try to unfreeze last few convolution layers, mabey help.But still, watch out BN. There are many talk about this problem of keras's transfer learning.