I am doing a semantic segmentation task using a supervised algorithm to classify image pixels into one class or the other (binary classification). I am trying several classifiers and feature combinations and computing accuracy, precision, recall, f1 score and auc roc score using kfold validation. (weighted and per class) I feed the classifier an image on the prediction stage and get an output of pixels classifications (binary mask)
I need to select the best classifier for the next stage. I'll explain.
After prediction I get a probably noisy image mask, then I use DBSCAN to cluster pixels and do some selection on them. Then I do a polynomial lasso around contours and some morphological transformations to close holes.
In stage 2 I compute jaccard index between prediction and ground-truth but I can't seem to find which classification metric would yield the best result in stage 1 that actually yield the best result in stage 2.