8 votes
Accepted

Micro Average vs Macro Average for Class Imbalance

The question is actually about understanding what it means to "take imbalance into account": Micro-average "takes imbalance into account" in the sense that the resulting ...
Erwan's user avatar
  • 25.3k
4 votes
Accepted

Is the PR AUC invariant under label flip?

I propose to look at this a bit differently. The different classification status of the two classes are related as follows: $TP_1=TN_0$ $FP_1=FN_0$ $FN_1=FP_0$ $TN_1=TP_0$ From there we have: $$...
Erwan's user avatar
  • 25.3k
4 votes

What is AUC - ROC Curve?

Related concepts, but not the same. ROC-receiver operating curve AUC area under the curve Thank this post for explanation : Abbreviations AUC = Area Under the Curve. AUROC = Area Under the ...
Noah Weber's user avatar
  • 5,669
3 votes

AUC-ROC for Multi-Label Classification

I am not sure if I understand your thinking, but my understanding is following - maybe it can help you see it from another perspective. The multi-label classification problem with ...
hans's user avatar
  • 253
3 votes

How much can the AUC improve comparing the raw dataset and the feature engineered dataset?

Yes, the performance can vary a lot using feature engineering. Example: suppose a dataset where the response variable $y$ is true if $x$ is odd. ...
Erwan's user avatar
  • 25.3k
3 votes

How to choose between different models with similar results? RF, GLM and XGBoost

Should I do more performance measurements? Which ones and why? Generally speaking, the choice of performance metric should be informed by your goals and the characteristics of your dataset. If you ...
zachdj's user avatar
  • 2,684
2 votes

Truncating float/doubles for reproducibility

I'm not expert in this but as far as I know the proper way to test for equality modulo floating point imprecision is to compare the differences of the two values, i.e. instead of: ...
Erwan's user avatar
  • 25.3k
2 votes

How much can the AUC improve comparing the raw dataset and the feature engineered dataset?

I would say that the best possible model for the raw data would derive all the meaningful features that you would have created from the data anyway. And I would say that the best possible model for ...
Pieter21's user avatar
  • 1,041
2 votes

Plotting ROC & AUC for SVM algorithm

If you are performing a binary classification task then the following code might help you. from sklearn.model_selection import GridSearchCV for hyper-parameter ...
Prasann's user avatar
  • 121
2 votes

Plotting ROC & AUC for SVM algorithm

The ROC curve requires probability estimates (or at least a realistic rank-ordering), which one-class SVM doesn't really try to produce. https://stats.stackexchange.com/a/99179/232706 https://...
Ben Reiniger's user avatar
  • 11.8k
2 votes

Confusion Matrix and AUC in univariate Anomaly Detection

It's difficult to answer precisely without knowing the data and the task. Assuming it's a single column of values with no order involved, it boils down to finding the optimal threshold to separate ...
Erwan's user avatar
  • 25.3k
2 votes

Don't understand why I get an inverse ROC curve for SVM (Python)

One potential fix is to remove max_iter = 12 (which would set it to the scikit learn default of max_iter=-1). Using such a low ...
Jonathan's user avatar
  • 5,410
2 votes

Don't understand why I get an inverse ROC curve for SVM (Python)

For any classification problem if AUC<0.5, you are not performing better than random(0.5). Reason could be: Your classifier is over-fitted on the training set and performs very poorly on the test ...
prashant0598's user avatar
  • 1,501
2 votes

How to compute AUC in gridsearchSV (multiclass problem)

There are actually several flavors of AUC you can now use with multiclass evaluation: 'roc_auc_ovo' 'roc_auc_ovo_weighted' 'roc_auc_ovr' 'roc_auc_ovr_weighted' These also work with ...
KT12's user avatar
  • 240
2 votes
Accepted

AUC ROC Curve multi class Classification

df = pd.get_dummies(pred1) df.insert(loc=2,column='2',value=0) #print(df) add this before the for loop and instead of using pd.get_dummies(y_test) use only df
Madhur Yadav's user avatar
2 votes

Does it make sense to repeat calculating AUC in logistic regression?

The calculation of ROC curve and the AUC based off of that curve is simply a comparison of the predictions from your model (logistic regression) and the actual values on some set of data. This can ...
engelAnalytics's user avatar
2 votes
Accepted

Overall AUC higher than all "stratified" AUCs

AUC can be defined as $P(X_1 > X_0)$ where $X_1$ is the score of a randomly chosen positive instance and $X_0$ is the score of a randomly chosen negative instance. Like in Simpson's "paradox&...
Valentas's user avatar
  • 1,209
2 votes
Accepted

Is roc auc graph better than roc auc score? If yes why?

Yes, the graph contains information that the AUC number alone does not have. It is most interesting when comparing 2+ models that have very close AUC numbers. The graph can tell you that one model ...
Darren Cook's user avatar
2 votes

Area Under the Precision Recall Curve

What happens is something like this: When the threshold is very high, with only very few instances predicted as positive, the precision is around 0.5. The recall is very low, since only a small ...
Erwan's user avatar
  • 25.3k
2 votes

Why is Precision-Recall AUC different from Average Precision score?

I have tried to generate a dataset with a 95:5 class imbalance, train a RandomForestClassifier model, and calculate AUPRC and AUC-ROC and Average Precision (AP) ...
Mario's user avatar
  • 400
1 vote

If ROC is used to find a threshold, but AUC is threshold invariant, why use AUC?

Roughly speaking, it depends on your purpose. AUC is for mathematical purpose (roughly speaking). It is a characteristic of the quality of your model. It depends on your data and your skills as model-...
lanenok's user avatar
  • 1,516
1 vote
Accepted

AUC higher than accuracy in multi-class problem

I think the question can se splitted to three parts. The first part is about comparing accuracy and AUC of the same model, the second is about comparing models and the third is about multi-class ...
Amit Keinan's user avatar
1 vote
Accepted

Implementing the Trapezoid rule without the formula for the curve

You're assuming that the points are equally spaced along the fpr axis, which is generally not true. See e.g. the "Uniform grid" vs "Nonuniform grid&...
Ben Reiniger's user avatar
  • 11.8k
1 vote

At what stage are ROC curves used when building machine learning model?

The ROC-AUC curves are used to find the best threshold that optimizes True Positive Rate vs False Positive Rate. Using it in a K-Fold cross-validation is a good practice to determine the best ...
etiennedm's user avatar
  • 1,395
1 vote
Accepted

Algorithm for Binary classification

First thing that comes to my mind is to do different encodings. There are some ways to deal with high cardinality categorical data such as: Label Encoding or the famous target encoding. Before ...
Carlos Mougan's user avatar
1 vote

How to compute AUC in gridsearchSV (multiclass problem)

Metrics are independent from ML algorithms, so it doesn't matter which algorithms did you use. To calculate multiclass AUC you could use lib pRoc in R or use code this link(in Python). Sources: ...
fuwiak's user avatar
  • 1,373
1 vote
Accepted

Confused AUC ROC score

Oh, I think I've finally got it. It's just an averaging problem: for each fold in your k-fold cross-validation, you get perfect auROC, but at the default threshold of 0.5 your hard classifiers (for ...
Ben Reiniger's user avatar
  • 11.8k
1 vote
Accepted

XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC)

With Success already being the larger class, you probably shouldn't be using a scale_pos_weight larger than one: you want to scale the positive class's contribution ...
Ben Reiniger's user avatar
  • 11.8k
1 vote

Main options on how to deal with imbalanced data

Imbalanced class means the count of one class is too low compared to the count of other Class. This means Model will have little opportunity to learn the minority Class. We have these option to ...
10xAI's user avatar
  • 5,584
1 vote

Main options on how to deal with imbalanced data

your resume is quite good, but I'm not comfortable in dividing the broad discussion to those three more or less sharply separated roads. But indeed, often a technique similar to one of those is chosen....
Vincenzo Lavorini's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible