# Confused AUC ROC score

I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .

def FPR(y_true, y_pred):
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
result = fp / (fp+tn)
return result
def FNR(y_true, y_pred):
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
result = fn / (tp+fn)
return result
FPR_scorer = make_scorer(FPR)
FNR_scorer = make_scorer(FNR)

def get_CrossValResults2(model,cv_rst,bestIndx):
best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]
roc="{:.12f}".format(best['mean_test_roc_auc'])
acc ="{:.0%}".format(best['mean_test_accuracy'])
prec ="{:.0%}".format(best['mean_test_precision'])
rec ="{:.0%}".format( best['mean_test_recall'])
f1 ="{:.0%}".format(best['mean_test_f1'])
r2="{:.2f}".format(best['mean_test_r2'])
g_mean="{:.2f}".format(best['mean_test_gmean'])
pr_auc="{:.8f}".format(best['mean_test_pr'])
fnr="{:.0%}".format(best['mean_test_fnr'])
fpr="{:.0%}".format(best['mean_test_fpr'])
rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])
return rst

cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])

scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}
param_grid = {'n_estimators': [200],
'max_depth': [80,90],
'min_samples_leaf': [2,3, 4],
'min_samples_split': [2,5,12],
'criterion': [ 'gini'],
'class_weight' : [class_weights], 'n_jobs' : [-1]}
clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model
bestmodel = clf.fit(X,Y)

cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)

• With only two decimals, it may just be rounding. Can you recover the actual value of ROC_auc? – Ben Reiniger Jul 20 '20 at 15:24
• cross-posted: stats.stackexchange.com/q/478046/232706 – Ben Reiniger Jul 20 '20 at 15:37
• The graphs you linked to in the comments to the answers are not very helpful. What about the actual ROC plots? What is the unrounded value of ROC_auc? – Ben Reiniger Jul 21 '20 at 17:52
• the values equals to 1.00000 imgur.com/6PmlMJ8 – Ak.tech Jul 21 '20 at 18:42
• It seems likely that something is wrong then... the ROC curve for RF is missing at least the .04 by .03 rectangle in the upper-left, so AUC should be at most 1-0.04*0.03=0.9988. Could you provide your code? – Ben Reiniger Jul 21 '20 at 23:37

Oh, I think I've finally got it. It's just an averaging problem: for each fold in your k-fold cross-validation, you get perfect auROC, but at the default threshold of 0.5 your hard classifiers (for each fold) sometimes have $$FPR=0$$ and $$TPR<1$$, but some other times $$FPR>0$$ and $$TPR=1$$. Then averaging you are able to get both $$\operatorname{mean}(FPR)>0$$ and $$\operatorname{mean}(TPR)<1$$.

To check, have a look at the cv_results_ table, particularly at each test fold scores (split<i>_test_<xyz>), rather than just the mean_test_<xyz> scores.

I think recall and FPR are calculated in scikit-learn using a threshold of 0.5. On the other hand ROC AUC is transparent to model threshold. I encourage you to explore thresholder in scikit-lego to inspect in this direction.

An example of AUC = 1 but bad FPR would be if you use 0.5 as a threshold, you model splits your samples perfectly but the positive ones have scores between 0.2 and 0.4 and your negative ones have scores between 0 and 0.2.

• That example just lands you in the corner of the ROC box though... – Ben Reiniger Jul 20 '20 at 15:23
• yes, I know, but what's the issue with the argument? – David Masip Jul 20 '20 at 15:40
• You're right that AUC=1 and "bad" FPR is possible, and the example works to that end, but the resulting point in ROC space isn't "bad"; the two corners will always be hit. (Actually, I think your example gives TPR=0, not FPR>0, but whatever. You could also construct an example with e.g. FPR=0 and TPR=0.5, etc.) To have AUC=1, you need every threshold to give a point in ROC space with either FPR=0 or TPR=1. You can't get both FPR>0 and TPR<1 and get perfect auROC as in OP, I think. – Ben Reiniger Jul 20 '20 at 15:45
• I need to write it down haha, thanks for your comment. I don't understand what you mean by bad point in ROC space, though. I just wanted to illustrate that you can have a good ROC curve but bad positive rate and negative metrics if a bad threshold is chosen – David Masip Jul 20 '20 at 15:49
• FPR and TPR values per thersholds RandomF =imgur.com/NsR2HEy AdaBoost=imgur.com/uD77mHj – Ak.tech Jul 20 '20 at 16:36