# Model not learning when using transfer learning

I am working on a personal project on image classification (two classes) and am trying to see how the MobileNet v2 structure would perform. While training the training accuracy is already quite high after the first epoch after which is continues to increase. The validation accuracy, however, seems forever stuck around the 50% mark.

I have tried both SGD, Adam, and RMSprop as optimizers, however they both produce the same results. I have also used both a MobileNet v2 with and without ImageNet weights without seeing any difference. Other CNN architectures work perfectly fine, achieving up to ~95% validation accuracy. I am using the following code for my model:

from keras.applications import MobileNetV2
from keras.models import Sequential
from keras.layers import GlobalAveragePooling2D, Dropout, Dense

basemodel = MobileNetV2(input_shape=(128, 173, 3), include_top=False, weights=None)
model = Sequential(
[
basemodel,
GlobalAveragePooling2D(),
Dropout(0.5),
Dense(64, activation="relu"),
Dropout(0.3),
Dense(1, activation="sigmoid"),
]
)
model.compile(loss="binary_crossentropy", optimizer="sgd", metrics=["accuracy"])


Is there something obvious that I am missing or am I just using incorrect hyperparameters for my dataset?

• In the MobileNetV2's constructor, you need to set weights="imagenet". – Shubham Panchal Nov 29 '19 at 0:43
• @ShubhamPanchal I also tried that, but it did not change anything with regards to the validation accuracy. – Oxbowerce Nov 30 '19 at 10:05