Questions tagged [auc]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Calculate AUC-ROC and AUC-PRC for an LSTM Model

I have the following simple Bidirectional LSTM model for a binary classification task: ...
thatsroughbuddy's user avatar
0 votes
0 answers
10 views

Understanding the code for calculating the 95% confidence interval of AUC using bootstrapping

It's really embarrassing, but I lack statistical knowledge. I would like to find the confidence interval for AUC at 95%. Actually, I got the code from here(https://stackoverflow.com/questions/52373318/...
JAE's user avatar
  • 13
0 votes
0 answers
12 views

PR-AUC vs F1 vs Balanced Accuracy

I'm trying to create a Random Forest Classifier for selecting ~ 700 features. I have a highly imbalanced dataset to select features from. There are significantly fewer positive cases (1%) compared ...
user155775's user avatar
1 vote
0 answers
20 views

How to Yield a Better AUC / Lift Score?

I have a dataset with 200k records and 173 features focused on binary classification. Class proportion is around 98.7:1.3 (1.3% target=1). Currently, I am trying to increase the performance of my ...
DM_FCP's user avatar
  • 11
1 vote
1 answer
126 views

Why is Precision-Recall AUC different from Average Precision score?

I have been calculating the area under the Precision-Recall curve (AUPRC) using the code snippet below: ...
chilipepper's user avatar
0 votes
0 answers
28 views

PR AUC curve with drop in precision

I have this PR AUC plot, with both PCA and autoencoder related curves having a huge drop of precision in the beginning and then increasing again, with PCA hitting 0 as you can see in the zoomed in ...
GabrielPast's user avatar
0 votes
0 answers
13 views

RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?

I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
Mig Rivera Cueva's user avatar
0 votes
0 answers
75 views

Mean Average Precision with 11 points interpolation method Python libs

I want to calculate mAP with 11 points interpolation method for object detection, as described here: https://learnopencv.com/mean-average-precision-map-object-detection-model-evaluation-metric/ What ...
Ars ML's user avatar
  • 61
0 votes
0 answers
18 views

AUC drops below 0.5 even though dataset stays similar

I'm programming an anomaly detection on a given dataset: Toyadmos dataset: https://arxiv.org/abs/1908.03299 Of this dataset, I'm investigating the ToyCar data, which has '4 cases': (quote)"Each “...
Brecht De Cock's user avatar
0 votes
1 answer
31 views

Overfitted model [duplicate]

A classic question with an unclear answer, is it better to have an overfitted model performing better on a Cross-Validation setting, or a non-overfitted model performing worse? In this context, higher ...
simon's user avatar
  • 133
0 votes
1 answer
199 views

How to explain relative difference between macro-AUC and macro-F1 in a multiclass classification problem?

I recently published a paper in which the result of a supervised model is the following. All the metrics are macro-averaged. I have been asked to comment on the gap between the AUC and the other ...
Eric Yamga's user avatar
0 votes
1 answer
130 views

Ugly AUC curves. Sklearn. How to make AUC Curves less square

I dislike the square look of this AUC curve (SKLearn). The purpose of this question is "visual". Please post code snippets. This question is not requesting the theory behind the AUC. My goal ...
Full Array's user avatar
1 vote
0 answers
119 views

What is ROC curve based on for SVM?

I was studying about the ROC curves for Logistic regression. There is a threshold in this method that determines the classification. By changing this threshold we get different confusion matrices and ...
Mina's user avatar
  • 111
7 votes
1 answer
267 views

How to choose between different models with similar results? RF, GLM and XGBoost

I am a medical doctor trying to make prediction models based on a database of approximately 1500 patients with 60+ parameters each. I am dealing with a classification problem (mortality at 1, 3, 6 and ...
user145725's user avatar
0 votes
0 answers
47 views

Video anomaly detection/ Evaluation AUC

I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array. scores = (scores - min(scores))/max(...
TecK97's user avatar
  • 1
2 votes
0 answers
80 views

Smallest possible difference between AUC of two ranker [closed]

If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
wrek's user avatar
  • 123
-1 votes
2 answers
1k views

Why I am having trouble plotting the AUC?

I am trying to plot the roc_auc curve however I am not getting any results. Any explanation here? Are there any problems with the number of data? Here is my example : ...
minattosama's user avatar
0 votes
2 answers
419 views

Should I be using y_pred or y_pred_proba for binary Classification?

I have a binary classification problem and i want to plot ROC/AUC curve, should I use ypred or ypred_proba
As13's user avatar
  • 77
1 vote
0 answers
29 views

How can I calculate de AUC PR of my classifiers in a multiclass scenario?

I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced. Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
Zaratruta's user avatar
  • 139
3 votes
1 answer
223 views

Area Under the Precision Recall Curve

I have got the following Precision Recall Curve for a classifier I built using AutoML. Most of the Precisio-Recall curves tend to start from (0, 1) go towards (1,0). But mine is the opposite. But I ...
user77005's user avatar
  • 151
1 vote
0 answers
16 views

My data can be approximated with Normal mixture. How can I find the reasons and explain this behaviour?

I use DeLonge method to compare two ROC AUCS. The result of it is Z-score. Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
Arzental's user avatar
1 vote
0 answers
17 views

How to ensamble different ranking models?

I have trained two different models, which give a score to each data point. The score of the models it is not necessarily comparable. The score is used to give a ranking, and the performance is ...
Diego Palacios's user avatar
0 votes
0 answers
28 views

Model Selection when there is trade-off

This is one of my model variants. It achieves an AUC score of 0.73. Another one of my model variants achieves an AUC score of 0.7265. Below is the the confusion matrix - Like many problems, the ...
a_jelly_fish's user avatar
1 vote
1 answer
14 views

Does thereshold of classifier close to 0 make sense?

I have roc curve with AUC of 0.91. I applied the following function to determine the best threshold: ...
Natili's user avatar
  • 13
3 votes
1 answer
4k views

Xgboost Multiclass evaluation Metrics

Im training an Xgb Multiclass problem, but im having doubts about my evaluation metrics, heres my code + output ...
Chichostyle's user avatar
0 votes
1 answer
127 views

What happens to auc when true positive rate grows

How does change in true positive rate affects AUC? Does increase of TPR lead to increase of AUC as well?
Ir8_mind's user avatar
  • 183
2 votes
3 answers
97 views

Is roc auc graph better than roc auc score? If yes why?

This was asked in viva of my ML course. I answered yes but could not precisely explain why. By 'better' I mean whether geometric interpretation gives more information than just the numeric score.
Rohit's user avatar
  • 23
1 vote
0 answers
56 views

Linear combination of features reverses importance of all features

I am trying with a logistic model with 2 features independently or with linear combination, but in the linear combination, combining these features would reverse importance through significance levels ...
user2842390's user avatar
0 votes
1 answer
354 views

Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier. Model Accuracy AUC Logistic Regression 0.671 0.736 Decision Tree (pruned) 0.681 0.665 Decision ...
khair's user avatar
  • 23
1 vote
1 answer
345 views

How to measure multi-label multi-class accuracy

I have a model that has multi-label multi-class targets Example Age Height Weight Mark Distance Red Yellow Green Blue Black White 14 160 62 78 103 0 1 1 1 1 0 56 177 90 99 363 1 1 0 0 0 0 32 179 ...
asmgx's user avatar
  • 549
1 vote
1 answer
47 views

How to interpret the Precision Recall AUC

The ROC AUC has an intuitive interpretation: the probability that the score of a randomly sampled 1-labeled item will be higher than a randomly sampled 0-labeled item. Is there a similar ...
Jacob G's user avatar
  • 121
0 votes
1 answer
804 views

Clarification on tf.keras.metrics.AUC

Question Does the AUC metric calculates the area of ROC or PR? Background tf.keras.metrics....
mon's user avatar
  • 711
1 vote
2 answers
154 views

If ROC is used to find a threshold, but AUC is threshold invariant, why use AUC?

Say I have a binary classifier. I calculate ROC to select an ideal threshold of say, 0.6. Then, I look at the AUC. But wait! If AUC doesn't change by selecting an 0.6 threshold, then what makes AUC ...
Monica Heddneck's user avatar
1 vote
1 answer
1k views

AUC higher than accuracy in multi-class problem

I stumbled upon a 3-class classification problem where all compared classifiers yield a higher AUC than accuracy (usually around 10% higher). This happens both when the dataset is balanced or slightly ...
razumichin's user avatar
0 votes
1 answer
229 views

Does it make sense to repeat calculating AUC in logistic regression?

I have a question regarding logistic regression models and testing its skill. I am not quite sure if I understand correctly how the ROC Curve is established. When calculating the ROC curve, is a train ...
DataVader's user avatar
1 vote
0 answers
48 views

What is the appropriate statistical test to compare the MAUC scores from two machine learning classifiers?

I would like to compare the scores of two multi-class classifiers. I have calculated the MAUC score for each of the algorithms, and now I want to see whether there is a statistical difference between ...
ILR's user avatar
  • 31
1 vote
0 answers
377 views

ROC and AUC curve for CNN multi-class classification problem

I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My ...
Jack's user avatar
  • 155
0 votes
0 answers
75 views

How to interpret stagnant validation curve

I'm new to deep learning, so I'm just learning how to interpret my models. I'm creating a mixed-convolutional neural net to classify melanoma images. Here's the model structure: ...
Yehuda's user avatar
  • 113
1 vote
1 answer
366 views

Overall AUC higher than all "stratified" AUCs

For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox. The AUC on the test set as a whole is 0.8. Gender is one of the model's features. So I decided to ...
Rohan Kadakia's user avatar
0 votes
1 answer
1k views

Implementing the Trapezoid rule without the formula for the curve

I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows: ...
David Stein's user avatar
3 votes
1 answer
293 views

At what stage are ROC curves used when building machine learning model?

When developing a machine learning model, at what stage are ROC curve with AUC used? Typically I have three data sets train - ...
erotavlas's user avatar
  • 130
1 vote
0 answers
392 views

Best practice to select precision vs. recall threshold for a binary classifier

I have a logistic regression model in Scikit-Learn doing a binary classification. Looking at the Roc curve for the classifier I can see that it performs really well: The AUC score is 0.99 which is ...
Sandy Lee's user avatar
  • 247
3 votes
2 answers
766 views

Is the PR AUC invariant under label flip?

The ROC-AUC curve is invariant under a flip of the labels. I don't know if its a famous result so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
Borun Chowdhury's user avatar
7 votes
1 answer
5k views

Micro Average vs Macro Average for Class Imbalance

I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional ...
machinery's user avatar
  • 246
4 votes
0 answers
197 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
Lucas Morin's user avatar
  • 2,196
2 votes
0 answers
2k views

AUC on ROC Curve near 1.0 for Multi-Class CNN but Precision/Recall are not perfect?

I am building a ROC Curve and calculating AUC for multi-class classification on the CIFAR-10 dataset using a CNN. My overall Accuracy is ~ 90% and my precision and recall are as follows: ...
Coldchain9's user avatar
0 votes
1 answer
726 views

AUC ROC Curve multi class Classification

Here is the part of the code for ROC AUC Curve calculation for multiple classes. ...
Rina's user avatar
  • 173
1 vote
1 answer
71 views

Algorithm for Binary classification

I have a data set with huge number of features ( Approximately 3000) and a binary target variable . The reason I have too many features is because of one hot encoding many categorical variables in ...
Bharathi A's user avatar
1 vote
0 answers
34 views

How to compute a confidence interval for AUC?

I found that in results of several binary classification problems, people report an AUC value together with a CI. I wonder how those CIs are computed. Is there a close-formed formula to compute them ...
Victor Luu's user avatar
2 votes
2 answers
3k views

Don't understand why I get an inverse ROC curve for SVM (Python)

I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves ...
MMMMMay's user avatar
  • 31