1
$\begingroup$

I am new to machine learning. This is my $1^{st}$ machine learning project and I am working on classification on an imbalanced dataset. There are also multi-classes in the target variable.

I would like to know what is the most suitable metrics for scoring the performance in the GridSearchCV.

I think

  1. roc_au is sometimes used for imbalanced dataset. But there are several

‘roc_auc’
‘roc_auc_ovo’
‘roc_auc_ovr’
Which should I use?

  1. Alternatively, precision-recall_auc is also used. But I can't seem to find this scoring metrics for GridSearchCV. How do I use it in GridSearchCV?

Thank you

X_train, X_test, y_train, y_test = train_test_split(X_total, Y_total, random_state=0, test_size=0.25)
kfold =GroupKFold(n_splits=3)
grid_search = GridSearchCV(RandomForestClassifier(random_state=0), hyperF, cv = kfold, scoring=, verbose = 1, n_jobs = -1)
$\endgroup$
1
  • $\begingroup$ Each of those scores can also just be specified as 'weighted', same with f1_score. $\endgroup$ Mar 12, 2022 at 14:37

3 Answers 3

0
$\begingroup$

One possible solution is to use scikit-learn's average_precision_score which is very similar to area under the precision-recall curve.

Since average_precision_score is a metric it will will work with scikit-learn's GridSearchCV.

$\endgroup$
0
$\begingroup$

Before performing Train Test Split, The most fundamental step for handling imbalanced data is to do UnderSampling or OverSampling , most of the SMOTE is what is recommended for the imbalaced data. you can use python package imblearn to do the SMOTE.

$\endgroup$
0
$\begingroup$

I would suggest first of all identifying your major and minor classes, identify which quantity out of True Positive, True Negative, False Positive and False Negative would you like to optimize and then choose the corresponding metric.

For the choice of metric I would suggest either going for Precision, Recall or F1 score (major or minor) depending on the quantity selected above.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.