Model not learning when using transfer learning

Question

I am working on a personal project on image classification (two classes) and am trying to see how the MobileNet v2 structure would perform. While training the training accuracy is already quite high after the first epoch after which is continues to increase. The validation accuracy, however, seems forever stuck around the 50% mark.

I have tried both SGD, Adam, and RMSprop as optimizers, however they both produce the same results. I have also used both a MobileNet v2 with and without ImageNet weights without seeing any difference. Other CNN architectures work perfectly fine, achieving up to ~95% validation accuracy. I am using the following code for my model:

from keras.applications import MobileNetV2
from keras.models import Sequential
from keras.layers import GlobalAveragePooling2D, Dropout, Dense

basemodel = MobileNetV2(input_shape=(128, 173, 3), include_top=False, weights=None)
model = Sequential(
    [
        basemodel,
        GlobalAveragePooling2D(),
        Dropout(0.5),
        Dense(64, activation="relu"),
        Dropout(0.3),
        Dense(1, activation="sigmoid"),
    ]
)
model.compile(loss="binary_crossentropy", optimizer="sgd", metrics=["accuracy"])

Is there something obvious that I am missing or am I just using incorrect hyperparameters for my dataset?

Answer 1

Well the critical step for finetuning/transfer learning any model in tensorflow[1,2+]:

• use right preprocessing for images (find the preprocessing which requires mobilenetv2)
• start with "imagenet" weights
• change the BatchNormalization behavior, in tf the default momentum value is 0.99 which is not best, in pytorch it is set to 0.9 which is better when finetunig your model and you don't have lot of data (see the link and this for more)

Answer 2

1. Use the imagenet weights.
2. Freeze the basemodel to prevent it from getting trained on your data.
3. You might be overfitting the trainset.Try to not use the whole MobileNet model, maybe only the first 10-15 layers.