5
$\begingroup$

I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights :

weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

then, in my network:

model.add(LSTM(..., class_weight=weights,...,callbacks=callbacks_list))

However, I also use a callbacks list to prevent overfitting, by imposing EarlyStopping based on validation_accuracy ...:

earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5, verbose=1, mode='auto')
callbacks_list = [earlystop]

However, my weights are only defined based on my training set... But my validation set also contains imbalanced data, in different proportions than the training set, and I would also like to give appropriate weights to have a fair evaluation of val_loss parameter...

So my question is

  1. With which weights is the val_loss computed ? With the weights given in class_weightparameter ? Or is this parameter only used for training loss ?

  2. How could I define weights that could be used for the validation set, to have a more accurate value of val_loss ?

Thanks

$\endgroup$
  • $\begingroup$ The weighting only happens during training time when we are passing class weights, Also if you have used Stratified Sampling and then split your data into train and validation, that should solve the problem..... Maybe Just another way around! $\endgroup$ – Aditya Sep 9 '18 at 3:41
  • $\begingroup$ Using validation set is a part of training process so must not be imbalanced. $\endgroup$ – Mehdi May 21 at 21:11
1
$\begingroup$

You are not supposed to apply weights to your validation set since it is supposed to measure your model's performance. If you'll do that you'll probably get better results for validation but once your model is deployed it will perform worse on new data. Weighting, resampling techniques etc. - they all should be done on the training set only!

| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't understand who marked this as -1 and why... $\endgroup$ – Corel Jun 1 at 17:31
0
$\begingroup$

As explained here you can define your custom metric. You only need to use validation set weight for calculating the score.

def my_metric(y_true, y_pred):
     sw=compute_sample_weight('balanced',y_true)
     return accuracy_score(y_true, y_pred, sample_weight=sw)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.