Questions tagged [auc]
The auc tag has no usage guidance.
65
questions
0
votes
0
answers
10
views
RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?
I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
0
votes
0
answers
43
views
Mean Average Precision with 11 points interpolation method Python libs
I want to calculate mAP with 11 points interpolation method for object detection, as described here:
https://learnopencv.com/mean-average-precision-map-object-detection-model-evaluation-metric/
What ...
- 61
0
votes
0
answers
18
views
AUC drops below 0.5 even though dataset stays similar
I'm programming an anomaly detection on a given dataset:
Toyadmos dataset: https://arxiv.org/abs/1908.03299
Of this dataset, I'm investigating the ToyCar data, which has '4 cases':
(quote)"Each “...
0
votes
1
answer
30
views
Overfitted model [duplicate]
A classic question with an unclear answer, is it better to have an overfitted model performing better on a Cross-Validation setting, or a non-overfitted model performing worse?
In this context, higher ...
- 133
0
votes
1
answer
73
views
How to explain relative difference between macro-AUC and macro-F1 in a multiclass classification problem?
I recently published a paper in which the result of a supervised model is the following.
All the metrics are macro-averaged.
I have been asked to comment on the gap between the AUC and the other ...
- 11
0
votes
1
answer
84
views
Ugly AUC curves. Sklearn. How to make AUC Curves less square
I dislike the square look of this AUC curve (SKLearn).
The purpose of this question is "visual". Please post code snippets.
This question is not requesting the theory behind the AUC.
My goal ...
- 156
1
vote
0
answers
90
views
What is ROC curve based on for SVM?
I was studying about the ROC curves for Logistic regression. There is a threshold in this method that determines the classification. By changing this threshold we get different confusion matrices and ...
- 111
6
votes
1
answer
187
views
How to choose between different models with similar results? RF, GLM and XGBoost
I am a medical doctor trying to make prediction models based on a database of approximately 1500 patients with 60+ parameters each.
I am dealing with a classification problem (mortality at 1, 3, 6 and ...
- 61
0
votes
0
answers
36
views
Video anomaly detection/ Evaluation AUC
I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array.
scores = (scores - min(scores))/max(...
- 1
2
votes
0
answers
77
views
Smallest possible difference between AUC of two ranker [closed]
If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
- 123
-1
votes
2
answers
756
views
Why I am having trouble plotting the AUC?
I am trying to plot the roc_auc curve however I am not getting any results. Any explanation here? Are there any problems with the number of data?
Here is my example :
...
- 204
0
votes
2
answers
354
views
Should I be using y_pred or y_pred_proba for binary Classification?
I have a binary classification problem and i want to plot ROC/AUC curve, should I use ypred or ypred_proba
- 47
1
vote
0
answers
26
views
How can I calculate de AUC PR of my classifiers in a multiclass scenario?
I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced.
Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
- 139
3
votes
1
answer
187
views
Area Under the Precision Recall Curve
I have got the following Precision Recall Curve for a classifier I built using AutoML. Most of the Precisio-Recall curves tend to start from (0, 1) go towards (1,0). But mine is the opposite. But I ...
- 151
1
vote
0
answers
16
views
My data can be approximated with Normal mixture. How can I find the reasons and explain this behaviour?
I use DeLonge method to compare two ROC AUCS. The result of it is Z-score.
Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
- 11
1
vote
0
answers
16
views
How to ensamble different ranking models?
I have trained two different models, which give a score to each data point. The score of the models it is not necessarily comparable.
The score is used to give a ranking, and the performance is ...
- 111
0
votes
0
answers
26
views
Model Selection when there is trade-off
This is one of my model variants. It achieves an AUC score of 0.73. Another one of my model variants achieves an AUC score of 0.7265. Below is the the confusion matrix -
Like many problems, the ...
- 197
1
vote
1
answer
14
views
Does thereshold of classifier close to 0 make sense?
I have roc curve with AUC of 0.91. I applied the following function to determine the best threshold: ...
- 13
3
votes
1
answer
3k
views
Xgboost Multiclass evaluation Metrics
Im training an Xgb Multiclass problem, but im having doubts about my evaluation metrics,
heres my code + output
...
- 31
0
votes
1
answer
100
views
What happens to auc when true positive rate grows
How does change in true positive rate affects AUC? Does increase of TPR lead to increase of AUC as well?
- 183
2
votes
3
answers
77
views
Is roc auc graph better than roc auc score? If yes why?
This was asked in viva of my ML course. I answered yes but could not precisely explain why.
By 'better' I mean whether geometric interpretation gives more information than just the numeric score.
- 23
1
vote
0
answers
51
views
Linear combination of features reverses importance of all features
I am trying with a logistic model with 2 features independently or with linear combination, but in the linear combination, combining these features would reverse importance through significance levels ...
- 13
0
votes
1
answer
294
views
Interpreting evaluation metrics with threshold/cutoff
I was doing churn prediction for a company. I've got the following results by applying 3 classifier.
Model
Accuracy
AUC
Logistic Regression
0.671
0.736
Decision Tree (pruned)
0.681
0.665
Decision ...
- 23
1
vote
1
answer
311
views
How to measure multi-label multi-class accuracy
I have a model that has multi-label multi-class targets
Example
Age
Height
Weight
Mark
Distance
Red
Yellow
Green
Blue
Black
White
14
160
62
78
103
0
1
1
1
1
0
56
177
90
99
363
1
1
0
0
0
0
32
179
...
- 539
1
vote
1
answer
42
views
How to interpret the Precision Recall AUC
The ROC AUC has an intuitive interpretation: the probability that the score of a randomly sampled 1-labeled item will be higher than a randomly sampled 0-labeled item.
Is there a similar ...
- 121
0
votes
1
answer
725
views
Clarification on tf.keras.metrics.AUC
Question
Does the AUC metric calculates the area of ROC or PR?
Background
tf.keras.metrics....
- 575
1
vote
2
answers
122
views
If ROC is used to find a threshold, but AUC is threshold invariant, why use AUC?
Say I have a binary classifier. I calculate ROC to select an ideal threshold of say, 0.6.
Then, I look at the AUC.
But wait! If AUC doesn't change by selecting an 0.6 threshold, then what makes AUC ...
- 477
1
vote
1
answer
1k
views
AUC higher than accuracy in multi-class problem
I stumbled upon a 3-class classification problem where all compared classifiers yield a higher AUC than accuracy (usually around 10% higher). This happens both when the dataset is balanced or slightly ...
- 23
0
votes
1
answer
170
views
Does it make sense to repeat calculating AUC in logistic regression?
I have a question regarding logistic regression models and testing its skill.
I am not quite sure if I understand correctly how the ROC Curve is established.
When calculating the ROC curve, is a train ...
- 15
1
vote
0
answers
39
views
What is the appropriate statistical test to compare the MAUC scores from two machine learning classifiers?
I would like to compare the scores of two multi-class classifiers. I have calculated the MAUC score for each of the algorithms, and now I want to see whether there is a statistical difference between ...
- 31
1
vote
0
answers
354
views
ROC and AUC curve for CNN multi-class classification problem
I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My ...
- 155
0
votes
0
answers
69
views
How to interpret stagnant validation curve
I'm new to deep learning, so I'm just learning how to interpret my models.
I'm creating a mixed-convolutional neural net to classify melanoma images.
Here's the model structure:
...
- 113
1
vote
1
answer
306
views
Overall AUC higher than all "stratified" AUCs
For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox. The AUC on the test set as a whole is 0.8.
Gender is one of the model's features. So I decided to ...
- 147
0
votes
1
answer
1k
views
Implementing the Trapezoid rule without the formula for the curve
I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows:
...
- 165
3
votes
1
answer
269
views
At what stage are ROC curves used when building machine learning model?
When developing a machine learning model, at what stage are ROC curve with AUC used?
Typically I have three data sets
train - ...
- 130
1
vote
0
answers
340
views
Best practice to select precision vs. recall threshold for a binary classifier
I have a logistic regression model in Scikit-Learn doing a binary classification.
Looking at the Roc curve for the classifier I can see that it performs really well:
The AUC score is 0.99 which is ...
- 247
3
votes
2
answers
650
views
Is the PR AUC invariant under label flip?
The ROC-AUC curve is invariant under a flip of the labels. I don't know if its a famous result so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
- 187
6
votes
1
answer
5k
views
Micro Average vs Macro Average for Class Imbalance
I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional ...
- 236
3
votes
0
answers
73
views
Fast PR / ROC curves and corespondings AUPR / AUROC
I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC).
Its is quite easy to perform those ...
- 2,567
2
votes
0
answers
2k
views
AUC on ROC Curve near 1.0 for Multi-Class CNN but Precision/Recall are not perfect?
I am building a ROC Curve and calculating AUC for multi-class classification on the CIFAR-10 dataset using a CNN. My overall Accuracy is ~ 90% and my precision and recall are as follows:
...
- 159
0
votes
1
answer
707
views
AUC ROC Curve multi class Classification
Here is the part of the code for ROC AUC Curve calculation for multiple classes.
...
- 173
1
vote
1
answer
67
views
Algorithm for Binary classification
I have a data set with huge number of features ( Approximately 3000) and a binary target variable . The reason I have too many features is because of one hot encoding many categorical variables in ...
- 75
1
vote
0
answers
28
views
How to compute a confidence interval for AUC?
I found that in results of several binary classification problems, people report an AUC value together with a CI. I wonder how those CIs are computed. Is there a close-formed formula to compute them ...
- 243
2
votes
2
answers
2k
views
Don't understand why I get an inverse ROC curve for SVM (Python)
I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves ...
- 31
0
votes
0
answers
20
views
confused AUC ROC score [duplicate]
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a 10 fold cross-validation ...
- 103
5
votes
2
answers
248
views
Confused AUC ROC score
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...).
I am using a cross validation technique (to ...
- 103
0
votes
1
answer
75
views
XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC)
My dataset is of shape – 5621*8 (binary classification)
Label/target : Success (4324, 77 %) & Not success (1297, 23 %)
(success and Not success were been ...
- 165
2
votes
2
answers
695
views
Main options on how to deal with imbalanced data
As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets:
Option 1:
Create k-fold Cross-Validation samples randomly (or even better create k-fold ...
- 21
1
vote
0
answers
73
views
Orange ROC analysis widget [closed]
ROC analysis widget has an option to change prior target class probability. I'd like to know, when and how it should be used. Playing with it changes the slope of iso-performance line. Each classifier ...
- 21
0
votes
2
answers
801
views
Getting lower performance metrics when using GridSearchCV
I have defined an XGBoost model and would like to tune some of its hyperparameters.
I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training ...
- 121