I'm building a logicsic regression classifier for binary classification. I have trained it and going to choose a cut-off value using ROC curve.
But what set to use for it: training or validation?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
Usually you need to generate the ROC curve and choose the threshold within the training data. Then, with the selected threshold, you have the possibility to report accuracy, sensitivity, recall results you reach on your validation set.
-
$\begingroup$ One more question. For example we build a logistic classifier, choose several values as the thresholds. So there are several classifiers with the same basis and different thresholds. Then we classify data points from the validation set and try to choose the best classifier. But it's equvalent to choosing the threshold. Is it correct? $\endgroup$ – Alexander Okunev Dec 26 '17 at 8:42
-
1$\begingroup$ I think the two approaches are similar. However by making a greedy search search of the best threshold on validation set, you risk to "learn" the validation set some way and select a not so optimal threshold due to data imperfections. $\endgroup$ – Theudbald Dec 26 '17 at 9:42
-
$\begingroup$ I guess my answer is partial and has to be completed. $\endgroup$ – Theudbald Dec 26 '17 at 9:42
