Initially I was evaluating my models using cross_val with out-of-pocket metrics such as precision, recall, f1 score, etc, or with my own metrics defined in make_scorer. The backbone of these metrics is the number of observation of TP, FP, TN, and FN from the confusion matrix. The evaluation results will simply look like this:

Fold 0, pre: 0.123, rec: 0.456, f1: 0.789; Fold 1, pre: ..., rec: ..., f1: ...; Fold 2 ...; Fold 3 ...; Fold 4 ...; Average: XXX

So by looking at the average scores I would know if this is the model I want, or I can set one metric as priority and proceed with hyperparameter tuning in GridSearchCV.

However, now I want my model to output predicted probability of each class (specifically when class = 1) instead of the actual class label, because in this case I would not need to worry about tuning class_weight given my super imbalanced data. (Please correct me if I have misunderstanding here.)

I am able to obtain the probability score for each data point by using cross_val_predict(clf, x, y, cv=5, method = 'predict_proba')

so I wonder how should I leverage this and proceed with model selection (e.g., using GridSerachCV) and model evaluation with some of the popular options and/or my own metrics after losing the confusion matrix.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.