About improving the classifier when using a pre-trained model

Question

I have tried adding a layer in the Resnet Model as shown:

fuente ="https://tfhub.dev/tensorflow/resnet_50/feature_vector/1"
class ResNetModel(tf.keras.Model):
    def __init__(self, classes):
        super(ResNetModel, self).__init__()        
        self._feature_extractor = hub.KerasLayer(fuente, trainable=False) 
        self._added_layer = tf.keras.layers.Dense(32, activation = "relu",trainable = True)
        self._classifier = tf.keras.layers.Dense(classes, activation='softmax')

    def call(self, inputs):
        x = self._feature_extractor(inputs)
        x= self._added_layer(x)
        x = self._classifier(x)
        return x

The test accuracy has diminished a lot. When training with more epochs, the training accuracy rapidly arrives at 100%, but the test accuracy stucks at 61.7%, much worse that using the model without the added layer(72% with 10 epochs). I expected some improvement in the result. Where is my error?

Answer 1

Both of your models (with and with no extra layer) have high train accuracy but much lower scores on test data which indicates that both models overfit, i.e. they suffer from high variance (using the bias-variance trade-off terminology). In essence, it means that your models perform poorly on unseen data.

To gain a better understanding of what is happening, I'd plot the learning curves of your model with no extra layer (see here for an explanation of and links to further reads on learning curves).

Generally, instead of adding an extra layer (and thereby increasing model capacity) you rather need to do the opposite in case of overfitting, e.g. by

• reducing model complexity (e.g. train for less epochs) or
• increasing the dataset size (e.g. by data augmentation).

The Deep Learning book by Ian Goodfellow provides some helpful practical recommendations in this chapter.