# Strange Neural Network overfitting

I'm experiencing a very strange behavior in training the following NN model for multiclass classification:

METRICS = [
keras.metrics.AUC(name='auc')
]

model = keras.Sequential()

for i in range(hidden_layers-1):

model.compile(
loss='categorical_crossentropy',
metrics=METRICS)


Testing different parameters combinations, my model always overfits, but the test accuracy is always lower than the train accuracy of 0.001, in terms of AUC. The following is an example of output obtained with 3 different combinations (learning rate, epochs, batch-size, hidden layers, and hidden units per each layer).

lr: 0.001, e: 10, b: 128, l: 1, u: 200
Train : 0.992
Test  : 0.991

lr: 0.001, e: 10, b: 128, l: 2, u: 200
Train : 0.984
Test  : 0.983

lr: 0.001, e: 10, b: 500, l: 1, u: 200
Train : 0.988
Test  : 0.987

lr: 0.001, e: 10, b: 500, l: 2, u: 200
Train : 0.974
Test  : 0.973


This is how I evaluate the model:

train_auc = roc_auc_score(y_train, model.predict(X_train), average='weighted')
test_auc = roc_auc_score(y_test, model.predict(X_test), average='weighted')


Please note that I'm using average='weighted' because I'm dealing with an imbalanced dataset.

I've tried training the model with and without Dropout and regularizers, but I've obtained the same strange behavior. What am I doing wrong?

• What is the accuracy of the Model on Minority Class? – 10xAI Jun 10 at 13:03
• @RoshanJha In the test set? The minority class has just 72 elements. – Mattia Campana Jun 10 at 20:22
• Could you describe the dataset you are working with? – Leevo Jun 19 at 7:55