3
$\begingroup$

I'm currently trying to train a RandomForestClassifier on a dataset consisting of 5000 instances with 12 (now) encoded features and a binary target label. Through GridSearchCV I found out, that

best_parameters = {
    'criterion':    'gini', 
    'max_depth':    12, 
    'max_features': 'log2', 
    'n_estimators': 300
}

works best out of

hyperparameters = {
    "n_estimators": [9,10,20,30,40,50,60,100,150,200,300,1000],
    "max_depth":    [3,6,9,12,20],
    "criterion":    ["gini", "entropy"],
    "max_features": ["log2", "auto"]
}

And gives a mean_test_score of 0.8546 which is quite good already, as I think.

Now I would like to get some kind of visual interpretation like a ROC curve for each parameter. But does it actually make sense in the case of a RandomForestClassifier to create a ROC curve for each hyperparameter? Or are there other ways to tune my classifier?

$\endgroup$
  • $\begingroup$ I don't think there is a threshold for activation for rf. Roc curves can be created for logistic regression, linear discriminate analysis and maybe neural networks $\endgroup$ – Darrin Thomas Oct 9 '17 at 14:00
  • $\begingroup$ I'm sorry, I don't quite understand what you mean. $\endgroup$ – herhuf Oct 9 '17 at 14:04
  • $\begingroup$ @DarrinThomas Random Forests do have a default threshold of 0.5 in that the majority vote 'wins'. This only works if the class probability is equal to the proportion of class votes, which it commonly is. Since this is not what rfs are designed to do, I don't know if it is legitimate to calculate probabilities in this way. $\endgroup$ – Eumenedies Oct 9 '17 at 15:35
1
$\begingroup$

I assume you are running classification, and have a binary target variable. If that's the case, it does not make sense to show component ROC curves, because your separation may be based on on combinations of 2, 3, or more predictors that individual ROC curves will not reflect. I would show your overall ROC curve, along with perhaps variable importance measures. If you have a handful of predictors that are clear winners, you could re-run your model including only those, and then show that ROC. Otherwise, I don't see what it buys you.

| improve this answer | |
$\endgroup$
3
$\begingroup$

Seems like you already tuned your algorithm based on the mean_test_score. You also could tune the AUC, which is the area under the ROC curve. Tuning this can even provide better mean_test_score as I show in my blog post: http://philipppro.github.io/Tuning_random_forest/

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.