I'm performing a classification of imbalanced multiclass data using a Neural Network in the TensorFlow framework. Therefore, I'm applying class weights.
I would like to apply early stopping to reduce overfitting. My concern is that the cost of the validation set used for early stopping will be calculated differently from the cost of the training set due to the class weights, so the early stopping will not work correctly. That's because the cost of the validation set could be biased to the classes that are over-represented in the data.
My questions are:
- Is the concern expressed above correct?
- If the answer is "yes", is it possible to apply class weights or sample weights on the validation set in TensorFlow in order that the cost of the training and the validation set will be calculated in a similar way?
- If it's not possible in TensorFlow, is it possible in PyTorch or other frameworks?
- Perhaps there are other solutions to the expressed concern?
My current relevant piece of code is:
model_checkpoint_callback = None checkpoint_filepath = r'C:\Users\User\PycharmProjects\models\SUAI\nn_checkpoint' if early_stopping: model_checkpoint_callback = ModelCheckpoint( filepath=checkpoint_filepath, save_weights_only=True, monitor='val_loss', mode='min', save_best_only=True ) history = nn.fit( X_train, y_train, validation_data=(X_val, y_val), epochs=n_epochs, callbacks=[model_checkpoint_callback] if early_stopping else None, batch_size=batch_size, verbose=verbose, class_weight=class_weights, sample_weight=sample_weight ) if early_stopping: nn.load_weights(checkpoint_filepath)