# Why are results without Transfer Learning better than with Transfer Learning?

I developed a neural network for license plate recognition and used the EfficientNet architecture (https://keras.io/api/applications/efficientnet/#efficientnetb0-function) with and without pretrained weights on ImageNet and with and without data augmentation. I only had 10.000 training images and 3.000 validation images. That was the reason I applied Transfer learning and image augmentation (AdditiveGaussianNoise).

I created this model:

efnB0_model = efn.EfficientNetB0(include_top=False, weights="imagenet", input_shape=(224, 224, 3))
efnB0_model.trainable = False

def create_model(input_shape = (224, 224, 3)):
input_img = Input(shape=input_shape)
model = efnB0_model (input_img)
model = GlobalAveragePooling2D(name='avg_pool')(model)
model = Dropout(0.2)(model)
backbone = model

branches = []
for i in range(7):
branches.append(backbone)
branches[i] = Dense(360, name="branch_"+str(i)+"_Dense_360")(branches[i])
branches[i] = BatchNormalization()(branches[i])
branches[i] = Activation("relu") (branches[i])
branches[i] = Dropout(0.2)(branches[i])
branches[i] = Dense(35, activation = "softmax", name="branch_"+str(i)+"_output")(branches[i])

output = Concatenate(axis=1)(branches)
output = Reshape((7, 35))(output)
model = Model(input_img, output)

return model


I compiled the model:

opt = keras.optimizers.Adam(learning_rate=0.0001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=["accuracy"])


And used this code to fit it:

hist = model.fit(
x=training_generator, epochs=10, verbose=1, callbacks=None,
validation_data=validation_generator, steps_per_epoch=num_train_samples // 16,
validation_steps=num_val_samples // 16,
max_queue_size=10, workers=6, use_multiprocessing=True)


My hypotheses were:

H1: The EfficientNet architecture is applicable to license plate recognition.

H2: Transfer learning will improve accuracy in license plate recognition (compared to the situation without Transfer Learning).

H3: Image augmentation will improve accuracy in license plate recognition (compared to the situation without it).

H4: Transfer Learning combined with Image augmentation will bring the best results.

I now got this results:

So, H1 seems to be correct. But H2, H3 and H4 seem to wrong.

I was thinking about it and got an explanation for H3 and H4, which seem to be logical for me. That is, that image augmentation is too heavy and deteriorates the quality of images by a degree which makes it very hard for the network to recognize the characters.

1. Is this a suitable explanation and are there other ones additionally?

It seems to be the case, that image augmentation was too strong. So, first question is solved.

Regarding H2 I am little confued to be honest. The network seems to overfit but stagnates completely regarding validation accuracy. So, the conclusion that the Imagenet weights are not applicable seems not logical to me because the network learnt something for the training data. I also excluded the possibility that the data volume is to small since we had that good recognition rates without using Transfer learning or image augmentation...

2. Is there any logical explanation for this?

• In the TL cases: did you only train the final layer? Generally, including your code would be useful. – Sammy Aug 3 '20 at 20:28
• I updated my question. So, I trained all the layers after model = efnB0_model (input_img). – Tobitor Aug 3 '20 at 21:57
• I agree this is unexpected. Could you share your dataset so we can reproduce those results and dig into their interpretation? Could you also share the other hyperparameters you used for training ? (optimizer, ...) – etiennedm Aug 6 '20 at 12:30
• Maybe we could do this via chat? I updated my question regarding the hyperparameters. – Tobitor Aug 6 '20 at 12:34
• @etiennedm: I could share my colab notebook and data with you via mail? – Tobitor Aug 6 '20 at 12:43

As @fuwiak mentioned, transfer learning may not work if pre-trained model has been fitted on a "very different" dataset. Typically if the pre-trained network extract information that is not relevant for your problem.

Moreover, in the paper License Plate Recognition System Based on Transfer Learning (that you shared with me), they have tried to freeze some layers of a pretrained Xception (based on ImageNet weights) to see the impact on the training. They conclude that ImageNet data and license plate data are too different to freeze layers. So your results are confirmed.

Now changing efnB0_model.trainable = False to True would allow the pre-trained network to update and to be more relevant to your problem. Generally, if you don't have time issues, it seems to be always better (see this post). Will it give better results than initialize the weights randomly ? I think one can presume but cannot know.

At least two issues:

Negative transfer

• Transfer learning working if the initial and our problem are similar. Unfortunately, we think that there are similar enough, but its just illusion.

Data greedy

• Often model start working well, if we provide much more data.
• Thank you! Regarding negative transfer: But how could the algorithm than learn the training data at all? – Tobitor Aug 6 '20 at 12:11
• To transfer learning often we are using State of art model. They are usually pretrained on specific dataset(in documentation on model should be specified this info). This one main advantages of transfer learning: usually we don't have access to good data and memory to fit model, then we using State of art model to way around these problems. – fuwiak Aug 6 '20 at 12:22
• Yeah, I understand this. But why could the model learn the training data and not the validation / test data? I suppose, if there is negative transfer learning, the model cannot predict neither training nor validation data, correct? – Tobitor Aug 6 '20 at 12:26
• @Tobitor, you could concatenate test and train set, shuffle connected set and use cross-validation. Then you will learn model on entire dataset. – fuwiak Aug 6 '20 at 12:34
• weight readjustments - Sometimes the weights transferred behave negatively which results in negative transfer and do not get rearranged with the new data. Check the weights before and after transfer and tweak it for learning purposes to get diff results. – Syenix Aug 6 '20 at 21:34