2
$\begingroup$

We are using random forest algorithm but having some trouble understanding the scoring method it uses.

take for example the following CM of the test set:

Threshold 45 cm is: 
[[67969 48031]
 [ 3321 11120]] and the prescion is: 0.18799344051632602
Threshold 50 cm is: 
[[77642 38358]
 [ 4785  9656]] and the prescion is: 0.2011080101632834
Threshold 55 cm is: 
[[88825 27175]
 [ 6796  7645]] and the prescion is: 0.2195577254445159
Threshold 60 cm is: 
[[100411  15589]
 [  9629   4812]] and the prescion is: 0.2358707906463611
Threshold 65 cm is: 
[[112421   3579]
 [ 13098   1343]] and the prescion is: 0.2728565623674755
Threshold 70 cm is: 
[[115895    105]
 [ 14371     70]] and the prescion is: 0.3999999997714286
Threshold 75 cm is: 
[[115998      2]
 [ 14440      1]] and the prescion is: 0.3333333222222226
Threshold 80 cm is: 
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0
Threshold 85 cm is: 
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0
Threshold 90 cm is: 
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0

This is how we used the RF and printed it's score:

grid_clf = RandomizedSearchCV(clf, param_grid, cv=tscv, verbose=10,n_iter=20,n_jobs=-1,scoring='roc_auc')
grid_clf.fit(X_train, y_train)
print(grid_clf.score(X_test,y_test))

The score we got for this model is 0.7350173458471928

As far as I understand the scoring when using roc_auc is between 0.5 and 1.

How can such a bad model received such a good score?

How is this scoring calculated?

Provided we predicted enough True Positives, we don't mind missing "1"'s and predicting False Positives. we of course do mind predicting True Negatives

Can I change the scoring to fit what I believe are better results?

Thanks

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Why do you say the model is bad and the AUROC is good? I have models where an AUROC of 0.95 can be improved and an AUROC of 0.65 is the best we can get. Also AUROC is a rank-order probabilistic measure. Make sure that fits the problem you are solving. Since you implied there are different costs to TP/FP/FN/TN, auroc might not be the best measure for you. stats.stackexchange.com/questions/180638/… $\endgroup$
    – Craig
    Mar 5, 2020 at 11:18
  • $\begingroup$ I say its bad because the precision is very low. I tried using the precision scoring metric when creating the model but it ended up with nearly not enough TT predictions. $\endgroup$
    – Amit Raz
    Mar 5, 2020 at 12:29
  • $\begingroup$ Maybe your precision measures are too distant from each other so you don't see the highest value? the prediction score seems to peak somewhere between thresholds 65 and 75, did you try to calculate it more precisely between these values (say every .5 for instance: 65.5, 61.0,...74.5, 75.0) $\endgroup$
    – Erwan
    Mar 5, 2020 at 16:00
  • $\begingroup$ First, do this report based on your benefits/costs. If a TP/FP/FN/TN have different benefits/costs as implied in the question, calc total benefit - cost instead of precision. Then it sounds like then a better model needs to be built, if possible, that satisfies the business needs. More features/data/engineering, different target definition, residual analysis, calibration and score histograms to understand where the model is weak, etc. Scikit has a bagging and voting ensemble classifier that might help to adjust the scoring mechanism. But first, I would match to the business and better model. $\endgroup$
    – Craig
    Mar 6, 2020 at 11:40

1 Answer 1

Reset to default
1
$\begingroup$

The evaluation metric used for classification .score is accuracy.

Your model has higher accuracy and lower precision. Looking at your confusion matrix, you have class imbalances which can lead to that result.

You should not change your evaluation metric to get better results. It is better to pick the evaluation that makes the most sense for the problem. Then improve the model through collecting more data, engineering better features, changing algorithms, or tuning hyperparameters.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.