5
$\begingroup$

I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights :

weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

then, in my network:

model.add(LSTM(..., class_weight=weights,...,callbacks=callbacks_list))

However, I also use a callbacks list to prevent overfitting, by imposing EarlyStopping based on validation_accuracy ...:

earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5, verbose=1, mode='auto')
callbacks_list = [earlystop]

However, my weights are only defined based on my training set... But my validation set also contains imbalanced data, in different proportions than the training set, and I would also like to give appropriate weights to have a fair evaluation of val_loss parameter...

So my question is

  1. With which weights is the val_loss computed ? With the weights given in class_weightparameter ? Or is this parameter only used for training loss ?

  2. How could I define weights that could be used for the validation set, to have a more accurate value of val_loss ?

Thanks

$\endgroup$
2
  • $\begingroup$ The weighting only happens during training time when we are passing class weights, Also if you have used Stratified Sampling and then split your data into train and validation, that should solve the problem..... Maybe Just another way around! $\endgroup$ – Aditya Sep 9 '18 at 3:41
  • $\begingroup$ Using validation set is a part of training process so must not be imbalanced. $\endgroup$ – Mehdi May 21 '20 at 21:11
1
$\begingroup$

You are not supposed to apply weights to your validation set since it is supposed to measure your model's performance. If you'll do that you'll probably get better results for validation but once your model is deployed it will perform worse on new data. Weighting, resampling techniques etc. - they all should be done on the training set only!

$\endgroup$
3
  • $\begingroup$ I don't understand who marked this as -1 and why... $\endgroup$ – Corel Jun 1 '20 at 17:31
  • $\begingroup$ This means I would define two different losses for calculating training and validation loss? Since training loss function would require me initializing weights that would be absent in case of validation $\endgroup$ – samra irshad Dec 11 '20 at 0:24
  • $\begingroup$ No. The loss function is the same. In validation you use your trained model on new data. "Trained" means that you apply learned coefficients on this new data. The information learned with the help of the weighting is already "embedded" in the coefficients. $\endgroup$ – Corel Dec 12 '20 at 7:45
0
$\begingroup$

As explained here you can define your custom metric. You only need to use validation set weight for calculating the score.

def my_metric(y_true, y_pred):
     sw=compute_sample_weight('balanced',y_true)
     return accuracy_score(y_true, y_pred, sample_weight=sw)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.