I'm experiencing a very strange behavior in training the following NN model for multiclass classification:
METRICS = [
keras.metrics.AUC(name='auc')
]
model = keras.Sequential()
model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1), input_shape=(input_len,)))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(dropout_rate))
for i in range(hidden_layers-1):
model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1)))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(dropout_rate))
model.add(keras.layers.Dense(output_len, activation="softmax"))
model.compile(
optimizer=tf.keras.optimizers.Adamax(learning_rate=lr),
loss='categorical_crossentropy',
metrics=METRICS)
Testing different parameters combinations, my model always overfits, but the test accuracy is always lower than the train accuracy of 0.001, in terms of AUC. The following is an example of output obtained with 3 different combinations (learning rate, epochs, batch-size, hidden layers, and hidden units per each layer).
lr: 0.001, e: 10, b: 128, l: 1, u: 200
Train : 0.992
Test : 0.991
lr: 0.001, e: 10, b: 128, l: 2, u: 200
Train : 0.984
Test : 0.983
lr: 0.001, e: 10, b: 500, l: 1, u: 200
Train : 0.988
Test : 0.987
lr: 0.001, e: 10, b: 500, l: 2, u: 200
Train : 0.974
Test : 0.973
This is how I evaluate the model:
train_auc = roc_auc_score(y_train, model.predict(X_train), average='weighted')
test_auc = roc_auc_score(y_test, model.predict(X_test), average='weighted')
Please note that I'm using average='weighted' because I'm dealing with an imbalanced dataset.
I've tried training the model with and without Dropout and regularizers, but I've obtained the same strange behavior. What am I doing wrong?