Strange Neural Network overfitting

Question

I'm experiencing a very strange behavior in training the following NN model for multiclass classification:

METRICS = [
    keras.metrics.AUC(name='auc')
]

model = keras.Sequential()
model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1), input_shape=(input_len,)))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(dropout_rate))

for i in range(hidden_layers-1):
    model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1)))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Dropout(dropout_rate))

model.add(keras.layers.Dense(output_len, activation="softmax"))

model.compile(
    optimizer=tf.keras.optimizers.Adamax(learning_rate=lr),
    loss='categorical_crossentropy',
    metrics=METRICS)

Testing different parameters combinations, my model always overfits, but the test accuracy is always lower than the train accuracy of 0.001, in terms of AUC. The following is an example of output obtained with 3 different combinations (learning rate, epochs, batch-size, hidden layers, and hidden units per each layer).

lr: 0.001, e: 10, b: 128, l: 1, u: 200
Train : 0.992
Test  : 0.991

lr: 0.001, e: 10, b: 128, l: 2, u: 200
Train : 0.984
Test  : 0.983

lr: 0.001, e: 10, b: 500, l: 1, u: 200
Train : 0.988
Test  : 0.987

lr: 0.001, e: 10, b: 500, l: 2, u: 200
Train : 0.974
Test  : 0.973

This is how I evaluate the model:

train_auc = roc_auc_score(y_train, model.predict(X_train), average='weighted')
test_auc = roc_auc_score(y_test, model.predict(X_test), average='weighted')

Please note that I'm using average='weighted' because I'm dealing with an imbalanced dataset.

I've tried training the model with and without Dropout and regularizers, but I've obtained the same strange behavior. What am I doing wrong?

Answer 1

I wouldn't say your model is overfitting, for two reasons:

• The validation AUC is extremely high, I don't think many models end up with ~0.98 AUC in validation. This essentially means that the model is perfectly learning the task given, with the possibility of some exceptions. These exceptions might be always a thing the same, for each training experiment you've done, and for this reason you see a consistent drop of performance.
• Whenever you train a model, it is normal for the validation performance to be slightly lower than the training performance, and that doesn't necessarily mean the model is overfitting, especially when the difference is around 1/1000 of AUC. I think this question might help you understand, for instance. I would say a model overfits if there is a significant drop, which I don't see here.