0
$\begingroup$

Is there any theoretical justification for using a different metric on the validation set to do model selection than what was using for training? For example, one can train the model using some type of differentiable loss (i.e. log-loss) so gradient descent can be ran, then do model selection using some other metric (i.e. F1 score or average precision). Is it the general hope that that the training metric is a good proxy for the validation metric, or that doing better on one means doing better on the other one?

$\endgroup$

1 Answer 1

0
$\begingroup$

Validation metric is selected based on the final goal of the model. For e.g., in medical image analysis to detect the presence of cancer, false negatives (detecting cancerous images as non cancer) should be very low as it is a disastrous condition, but some false positives (detecting non cancerous images as cancer) is still OK as it will not be very harmful or fatal. So, in this case, we tend to focus more on keeping recall(TP/TP+FN ) high to get high TP when compared with FN. Since recall score is non differentiable (cannot be used as loss function), we tend to incorporate it in our loss function which was used for training the model, for better results. Go through this article: https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d

" Is it the general hope that that the training metric is a good proxy for the validation metric?" Well, yes. The purpose of training the model is to enable it to perform the desired task on unseen data (evaluated using validation metric). Through training, the model learns what to do on new data (validation data), it will perform what it will learn, so to achieve good results on validation data, training metric should be a good proxy for the validation metric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.