Class leaking on validation set

Question

I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements that I will now report. This discussion will be based on Keras only for practical reasons.

1. First of all: Keras initially supported various metrics now removed, like for example precision, recall, f1, fbeta. As stated here they were removed only for a clarity reason: they were calculated as local metrics and so they were (almost) meaningless.
2. Those metrics clearly used the labels/classes to be computed. However in that post nobody complained about this problem
3. Now comes the hard part: Keras allows to specify both class and sample weights during the training (with the class_weight and sample_weight arguments). These both influence the training loss and the metrics listed in weighted_metrics.
4. However, class_weight does not influence the metrics listed in weighted_metrics during the validation nor can be used in the evaluate method, where it is only possibile to specify the sample_weight argument but not the class_weight one.

François Chollet clearly explained this design choice (quote):

You'll notice that evaluate has a sample_weight argument but no class_weight argument. Validation evaluation in fit follows the same pattern. The reason is that class weighting is meant to help training (by modulating contributions of different classes), but not to affect evaluation. Evaluation, because it reflects production constraints, implies that labels are not available when making predictions, and thus can't be used to modulate evaluation metrics. (link)

---

Additionally I have noticed that in the case where validation_split is used, class_weight is applied to the validation data, which is not the expected behavior (classes are of course supposed unknown for the validation data). I am also fixing this. (link)

---

I mean that weighting by classes in a test setup is a form of label information leak, because it cannot be reproduced in a real test situation (when you don't have the labels, such as when using model.predict()). It's acceptable during training, by definition, but should not be done at testing time. (link)

---

I cannot understand why this is the case during the validation phase:

• A class leaking like this is not such a serious problem: there is no backprop during the validation phase and so the potential leaking is quite limited. In addition I think this is not more serious than the standard leaking due to the continuous model tuning on the same validation set
• Without using the classes, during the validation we could not compute interesting quantities such as the precision, the recall, the confusion matrix and so on. Computing them on the training set would produce a biased estimate.
• Using the classes info would be thus extremely useful in order to obtain good estimates.
• In addition, as stated before, those metrics were available and nobody complained about them.

I know we could obtain a similar effect by playing with the sample_weight argument but I am more interested in understanding this problem. I know that the classes will not be available (theoretically) during the testing phase but I think that, during the development and tuning, these information and statistics could help better understanding the errors done by the model.

Answer 1

You could calculate these metrics and then attempt to clarify how your model is failing during validation, but then you're effectively turning your simulated test environment into a second round of training. Any further tuning decisions made from calculating these metrics defeat the purpose of using the validation set as a chance to prove your model's ability to generalize.