Questions tagged [auc]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Video anomaly detection/ Evaluation AUC

I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array. scores = (scores - min(scores))/max(...
  • 1
2 votes
0 answers
65 views

Smallest possible difference between AUC of two ranker [closed]

If there are 10 positive examples, and 90 negative examples in the test set, what is the smallest possible difference in AUC, between two rankers giving different AUC?
  • 121
0 votes
0 answers
163 views

AUC OvO vs AUC OvR vs F1-Score in Multiclass Model Selection, what is better?

Given a multiclass classification task, I am looking at the best metric between AUC OvO, AUC OvR and F1-Score score to compare models. The class distribution is the following (there is no 'class ...
0 votes
0 answers
16 views

Cause of randomness in AUC score for GNN

I have implemented a GraphSAGE model using dgl for link prediction. On average the auc score of the model is ~0.7 but the score varies a lot for different runs. ...
-1 votes
2 answers
105 views

Why I am having trouble plotting the AUC?

I am trying to plot the roc_auc curve however I am not getting any results. Any explanation here? Are there any problems with the number of data? Here is my example : ...
0 votes
0 answers
21 views

Why is ROC-AUC usually shown in GNN papers

In various graph neural network (GNNs) papers, the ROC-AUC metric is usually shown alone without considering F1 or Accuracy. Is there a reason for that? What does it say about two models 1 and 2 with ...
0 votes
2 answers
156 views

Should I be using y_pred or y_pred_proba for binary Classification?

I have a binary classification problem and i want to plot ROC/AUC curve, should I use ypred or ypred_proba
  • 47
0 votes
0 answers
10 views

How are ROC curves constructed? [duplicate]

I would like to understand how to build a ROC curve of a model. For example, if we would like to draw it by hand, what steps should we do? Thank you.
  • 302
1 vote
0 answers
23 views

How can I calculate de AUC PR of my classifiers in a multiclass scenario?

I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced. Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
  • 127
3 votes
1 answer
128 views

Area Under the Precision Recall Curve

I have got the following Precision Recall Curve for a classifier I built using AutoML. Most of the Precisio-Recall curves tend to start from (0, 1) go towards (1,0). But mine is the opposite. But I ...
  • 151
1 vote
0 answers
16 views

My data can be approximated with Normal mixture. How can I find the reasons and explain this behaviour?

I use DeLonge method to compare two ROC AUCS. The result of it is Z-score. Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
1 vote
0 answers
16 views

How to ensamble different ranking models?

I have trained two different models, which give a score to each data point. The score of the models it is not necessarily comparable. The score is used to give a ranking, and the performance is ...
0 votes
0 answers
31 views

Fashion Compatibility Performance Evaluation: High in AUC but Low in FITB

I am a newbie in deep learning field. Still trying to understand how this works. But now I am working on fashion compatibility prediction. The most well-known performance evaluation in this task is ...
  • 1
0 votes
0 answers
23 views

Model Selection when there is trade-off

This is one of my model variants. It achieves an AUC score of 0.73. Another one of my model variants achieves an AUC score of 0.7265. Below is the the confusion matrix - Like many problems, the ...
  • 167
1 vote
1 answer
12 views

Does thereshold of classifier close to 0 make sense?

I have roc curve with AUC of 0.91. I applied the following function to determine the best threshold: ...
  • 13
2 votes
1 answer
2k views

Xgboost Multiclass evaluation Metrics

Im training an Xgb Multiclass problem, but im having doubts about my evaluation metrics, heres my code + output ...
0 votes
1 answer
57 views

What happens to auc when true positive rate grows

How does change in true positive rate affects AUC? Does increase of TPR lead to increase of AUC as well?
  • 163
2 votes
3 answers
60 views

Is roc auc graph better than roc auc score? If yes why?

This was asked in viva of my ML course. I answered yes but could not precisely explain why. By 'better' I mean whether geometric interpretation gives more information than just the numeric score.
  • 23
1 vote
0 answers
37 views

Linear combination of features reverses importance of all features

I am trying with a logistic model with 2 features independently or with linear combination, but in the linear combination, combining these features would reverse importance through significance levels ...
0 votes
1 answer
178 views

Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier. Model Accuracy AUC Logistic Regression 0.671 0.736 Decision Tree (pruned) 0.681 0.665 Decision ...
  • 23
1 vote
1 answer
207 views

How to measure multi-label multi-class accuracy

I have a model that has multi-label multi-class targets Example Age Height Weight Mark Distance Red Yellow Green Blue Black White 14 160 62 78 103 0 1 1 1 1 0 56 177 90 99 363 1 1 0 0 0 0 32 179 ...
  • 517
1 vote
1 answer
31 views

How to interpret the Precision Recall AUC

The ROC AUC has an intuitive interpretation: the probability that the score of a randomly sampled 1-labeled item will be higher than a randomly sampled 0-labeled item. Is there a similar ...
  • 111
0 votes
1 answer
519 views

Clarification on tf.keras.metrics.AUC

Question Does the AUC metric calculates the area of ROC or PR? Background tf.keras.metrics....
  • 533
1 vote
2 answers
78 views

If ROC is used to find a threshold, but AUC is threshold invariant, why use AUC?

Say I have a binary classifier. I calculate ROC to select an ideal threshold of say, 0.6. Then, I look at the AUC. But wait! If AUC doesn't change by selecting an 0.6 threshold, then what makes AUC ...
1 vote
1 answer
680 views

AUC higher than accuracy in multi-class problem

I stumbled upon a 3-class classification problem where all compared classifiers yield a higher AUC than accuracy (usually around 10% higher). This happens both when the dataset is balanced or slightly ...
0 votes
1 answer
108 views

Does it make sense to repeat calculating AUC in logistic regression?

I have a question regarding logistic regression models and testing its skill. I am not quite sure if I understand correctly how the ROC Curve is established. When calculating the ROC curve, is a train ...
1 vote
0 answers
30 views

What is the appropriate statistical test to compare the MAUC scores from two machine learning classifiers?

I would like to compare the scores of two multi-class classifiers. I have calculated the MAUC score for each of the algorithms, and now I want to see whether there is a statistical difference between ...
  • 31
1 vote
0 answers
275 views

ROC and AUC curve for CNN multi-class classification problem

I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My ...
  • 145
0 votes
0 answers
47 views

How to interpret stagnant validation curve

I'm new to deep learning, so I'm just learning how to interpret my models. I'm creating a mixed-convolutional neural net to classify melanoma images. Here's the model structure: ...
  • 113
1 vote
1 answer
195 views

Overall AUC higher than all "stratified" AUCs

For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox. The AUC on the test set as a whole is 0.8. Gender is one of the model's features. So I decided to ...
0 votes
1 answer
667 views

Implementing the Trapezoid rule without the formula for the curve

I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows: ...
3 votes
1 answer
226 views

At what stage are ROC curves used when building machine learning model?

When developing a machine learning model, at what stage are ROC curve with AUC used? Typically I have three data sets train - ...
  • 130
1 vote
0 answers
212 views

Best practice to select precision vs. recall threshold for a binary classifier

I have a logistic regression model in Scikit-Learn doing a binary classification. Looking at the Roc curve for the classifier I can see that it performs really well: The AUC score is 0.99 which is ...
  • 217
3 votes
2 answers
376 views

Is the PR AUC invariant under label flip?

The ROC-AUC curve is invariant under a flip of the labels. I don't know if its a famous result so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
6 votes
1 answer
3k views

Micro Average vs Macro Average for Class Imbalance

I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional ...
  • 236
3 votes
0 answers
67 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
  • 2,320
2 votes
0 answers
1k views

AUC on ROC Curve near 1.0 for Multi-Class CNN but Precision/Recall are not perfect?

I am building a ROC Curve and calculating AUC for multi-class classification on the CIFAR-10 dataset using a CNN. My overall Accuracy is ~ 90% and my precision and recall are as follows: ...
0 votes
1 answer
652 views

AUC ROC Curve multi class Classification

Here is the part of the code for ROC AUC Curve calculation for multiple classes. ...
  • 140
1 vote
1 answer
62 views

Algorithm for Binary classification

I have a data set with huge number of features ( Approximately 3000) and a binary target variable . The reason I have too many features is because of one hot encoding many categorical variables in ...
1 vote
0 answers
21 views

How to compute a confidence interval for AUC?

I found that in results of several binary classification problems, people report an AUC value together with a CI. I wonder how those CIs are computed. Is there a close-formed formula to compute them ...
2 votes
2 answers
2k views

Don't understand why I get an inverse ROC curve for SVM (Python)

I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves ...
  • 31
0 votes
0 answers
20 views

confused AUC ROC score [duplicate]

I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a 10 fold cross-validation ...
  • 103
5 votes
2 answers
206 views

Confused AUC ROC score

I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a cross validation technique (to ...
  • 103
0 votes
1 answer
54 views

XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC)

My dataset is of shape – 5621*8 (binary classification) Label/target : Success (4324, 77 %) & Not success (1297, 23 %) (success and Not success were been ...
  • 165
2 votes
2 answers
384 views

Main options on how to deal with imbalanced data

As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets: Option 1: Create k-fold Cross-Validation samples randomly (or even better create k-fold ...
  • 21
1 vote
0 answers
67 views

Orange ROC analysis widget [closed]

ROC analysis widget has an option to change prior target class probability. I'd like to know, when and how it should be used. Playing with it changes the slope of iso-performance line. Each classifier ...
  • 21
0 votes
2 answers
588 views

Getting lower performance metrics when using GridSearchCV

I have defined an XGBoost model and would like to tune some of its hyperparameters. I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training ...
  • 121
1 vote
3 answers
1k views

How to compute AUC in gridsearchSV (multiclass problem)

I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the ...
  • 13
1 vote
1 answer
408 views

Confusion matrix and ROC AUC curves are not in sync

I created a classification model with three target classes and created a confusion matrix to measure the accuracy, here is the matrix code ...
-1 votes
2 answers
16k views

Plotting ROC & AUC for SVM algorithm

Towards , the end of my program, I have the following code. model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00001) model.fit(train_data) Output ...
  • 585