5
$\begingroup$

I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights :

weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

then, in my network:

model.add(LSTM(..., class_weight=weights,...,callbacks=callbacks_list))

However, I also use a callbacks list to prevent overfitting, by imposing EarlyStopping based on validation_accuracy ...:

earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5, verbose=1, mode='auto')
callbacks_list = [earlystop]

However, my weights are only defined based on my training set... But my validation set also contains imbalanced data, in different proportions than the training set, and I would also like to give appropriate weights to have a fair evaluation of val_loss parameter...

So my question is

  1. With which weights is the val_loss computed ? With the weights given in class_weightparameter ? Or is this parameter only used for training loss ?

  2. How could I define weights that could be used for the validation set, to have a more accurate value of val_loss ?

Thanks

$\endgroup$
2
  • $\begingroup$ The weighting only happens during training time when we are passing class weights, Also if you have used Stratified Sampling and then split your data into train and validation, that should solve the problem..... Maybe Just another way around! $\endgroup$
    – Aditya
    Commented Sep 9, 2018 at 3:41
  • $\begingroup$ Using validation set is a part of training process so must not be imbalanced. $\endgroup$
    – Mehdi
    Commented May 21, 2020 at 21:11

2 Answers 2

1
$\begingroup$

You are not supposed to apply weights to your validation set since it is supposed to measure your model's performance. If you'll do that you'll probably get better results for validation but once your model is deployed it will perform worse on new data. Weighting, resampling techniques etc. - they all should be done on the training set only!

$\endgroup$
3
  • $\begingroup$ I don't understand who marked this as -1 and why... $\endgroup$
    – Corel
    Commented Jun 1, 2020 at 17:31
  • $\begingroup$ This means I would define two different losses for calculating training and validation loss? Since training loss function would require me initializing weights that would be absent in case of validation $\endgroup$ Commented Dec 11, 2020 at 0:24
  • $\begingroup$ No. The loss function is the same. In validation you use your trained model on new data. "Trained" means that you apply learned coefficients on this new data. The information learned with the help of the weighting is already "embedded" in the coefficients. $\endgroup$
    – Corel
    Commented Dec 12, 2020 at 7:45
0
$\begingroup$

As explained here you can define your custom metric. You only need to use validation set weight for calculating the score.

def my_metric(y_true, y_pred):
     sw=compute_sample_weight('balanced',y_true)
     return accuracy_score(y_true, y_pred, sample_weight=sw)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.