VGG16 Transfer Learning for image binary classification - suspected overfitting

Question

I'm using VGG16 for transfer learning on a binary image classification task about human posture. The sample totaled about 2,000 images, with about 900 and 1,000 images in each category, respectively. The sample was labeled by separate human labelers. The accuracy of the validation set has been hovering at 65% while the accuracy of the training set is still going up. So I suspect that overfitting is occurring.

I have tried to tweak the dropout layer, learning rate (learning rate becomes too small performance becomes poor), batch size etc. I would like to know what is the problem? How can I improve the model accuracy?

PS: Most of the time, when I look at the confusion matrix, one category has a high precision and a low recall, and another category has just the opposite. I can't resolve it.

# Configuration of the data generator
datagen = ImageDataGenerator(
    horizontal_flip=True,
    zoom_range=0.2,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    rescale=1./255  
)

# validation set data generator
validation_generator = datagen.flow_from_dataframe(
    dataframe=val_df,
    x_col='image_path',
    y_col='category',
    target_size=(224, 224),
    batch_size=8,
    class_mode='binary',
    shuffle=False,
    seed=21
)

# training set data generator
train_generator = datagen.flow_from_dataframe(
    dataframe=train_df,
    x_col='image_path',
    y_col='category',
    target_size=(224, 224),
    batch_size=8,
    class_mode='binary',
    shuffle=True,
    seed=21
)

# Load the VGG16 model, include_top=False
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

# Freeze the first four convolutional blocks
for layer in base_model.layers[:19]:
    layer.trainable = False

# Add custom layers to the pre-trained model
x = base_model.output
x = Flatten()(x)
x = Dropout(0.75)(x)
x = Dense(128, activation='relu', kernel_regularizer=l2(0.1))(x)
output_layer = Dense(1, activation='sigmoid')(x)

# Create the final model
model = Model(inputs=base_model.input, outputs=output_layer)

# Adadelta
custom_adadelta_optimizer = Adadelta(learning_rate=0.0001)

# compile the model
model.compile(optimizer=custom_adadelta_optimizer, loss='binary_crossentropy', metrics=['accuracy'])

# print summary
model.summary()

# early stop
early_stopping = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

# train the model
history = model.fit_generator(
    train_generator,
    steps_per_epoch=len(train_generator),
    epochs=18,
    validation_data=validation_generator,
    validation_steps=len(validation_generator),
    callbacks=[early_stopping]
)
```

Answer 1

I don't know where you took the VGG16 from (HuggingFace has no VGG16 class as far as I know), but it looks like you didn't remove the VGG softmax layer before adding your additional layers, which is one possible reason for the results you are getting. Also, since you didn't mention it - have you tried multiple train/test splits or are all your experiments performed on the same split/seed?