1
$\begingroup$

I'm experiencing a very strange behavior in training the following NN model for multiclass classification:

METRICS = [
    keras.metrics.AUC(name='auc')
]

model = keras.Sequential()
model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1), input_shape=(input_len,)))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(dropout_rate))

for i in range(hidden_layers-1):
    model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1)))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Dropout(dropout_rate))

model.add(keras.layers.Dense(output_len, activation="softmax"))

model.compile(
    optimizer=tf.keras.optimizers.Adamax(learning_rate=lr),
    loss='categorical_crossentropy',
    metrics=METRICS)

Testing different parameters combinations, my model always overfits, but the test accuracy is always lower than the train accuracy of 0.001, in terms of AUC. The following is an example of output obtained with 3 different combinations (learning rate, epochs, batch-size, hidden layers, and hidden units per each layer).

lr: 0.001, e: 10, b: 128, l: 1, u: 200
Train : 0.992
Test  : 0.991

lr: 0.001, e: 10, b: 128, l: 2, u: 200
Train : 0.984
Test  : 0.983

lr: 0.001, e: 10, b: 500, l: 1, u: 200
Train : 0.988
Test  : 0.987

lr: 0.001, e: 10, b: 500, l: 2, u: 200
Train : 0.974
Test  : 0.973

This is how I evaluate the model:

train_auc = roc_auc_score(y_train, model.predict(X_train), average='weighted')
test_auc = roc_auc_score(y_test, model.predict(X_test), average='weighted')

Please note that I'm using average='weighted' because I'm dealing with an imbalanced dataset.

I've tried training the model with and without Dropout and regularizers, but I've obtained the same strange behavior. What am I doing wrong?

$\endgroup$
3
  • $\begingroup$ What is the accuracy of the Model on Minority Class? $\endgroup$
    – 10xAI
    Jun 10, 2020 at 13:03
  • $\begingroup$ @RoshanJha In the test set? The minority class has just 72 elements. $\endgroup$ Jun 10, 2020 at 20:22
  • $\begingroup$ Could you describe the dataset you are working with? $\endgroup$
    – Leevo
    Jun 19, 2020 at 7:55

1 Answer 1

1
$\begingroup$

I wouldn't say your model is overfitting, for two reasons:

  • The validation AUC is extremely high, I don't think many models end up with ~0.98 AUC in validation. This essentially means that the model is perfectly learning the task given, with the possibility of some exceptions. These exceptions might be always a thing the same, for each training experiment you've done, and for this reason you see a consistent drop of performance.
  • Whenever you train a model, it is normal for the validation performance to be slightly lower than the training performance, and that doesn't necessarily mean the model is overfitting, especially when the difference is around 1/1000 of AUC. I think this question might help you understand, for instance. I would say a model overfits if there is a significant drop, which I don't see here.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.