Questions tagged [auc]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

What is the appropriate statistical test to compare the MAUC scores from two machine learning classifiers?

I would like to compare the scores of two multi-class classifiers. I have calculated the MAUC score for each of the algorithms, and now I want to see whether there is a statistical difference between ...
0
votes
0answers
12 views

How to optimize AUPRC for imbalanced data given a precision or recall bias?

My general understanding is that when optimizing a model in an imbalanced class case with a small preferred target class one should optimize first for a model with the best AUPRC (assuming one doesn't ...
1
vote
0answers
22 views

ROC and AUC curve for CNN multi-class classification problem

I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My ...
0
votes
0answers
20 views

How to interpret stagnant validation curve

I'm new to deep learning, so I'm just learning how to interpret my models. I'm creating a mixed-convolutional neural net to classify melanoma images. Here's the model structure: ...
1
vote
1answer
55 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed Recursive Feature Elimination (RFE) and stratified cross validation to reduce the features to 15 and I get ...
0
votes
0answers
8 views

Model performance in different snapshots varying

I am trying to solve this problem. A medical representative needs to visit some doctors' clinics and for that a model will generate probability scores for visiting a clinic. I ma using a tree based ...
0
votes
0answers
9 views

LightGBM model improvement when the focus is on probability prediction

I am building a binary classifier using LightGBM. The goal is not to predict the outcome as such, but rather to predict the probability of the target even. To be more specific, it's more about ranking ...
1
vote
1answer
59 views

Overall AUC higher than all “stratified” AUCs

For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox. The AUC on the test set as a whole is 0.8. Gender is one of the model's features. So I decided to ...
0
votes
1answer
47 views

Implementing the Trapezoid rule without the formula for the curve

I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows: ...
2
votes
2answers
99 views

Main options on how to deal with imbalanced data

As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets: Option 1: Create k-fold Cross-Validation samples randomly (or even better create k-fold ...
3
votes
1answer
62 views

At what stage are ROC curves used when building machine learning model?

When developing a machine learning model, at what stage are ROC curve with AUC used? Typically I have three data sets train - ...
1
vote
0answers
66 views

Best practice to select precision vs. recall threshold for a binary classifier

I have a logistic regression model in Scikit-Learn doing a binary classification. Looking at the Roc curve for the classifier I can see that it performs really well: The AUC score is 0.99 which is ...
0
votes
0answers
20 views

When to use AUROC OvR vs. AUROC OvO?

For multiclass problems, there are 2 versions of the AUROC metric: the AUROC OvR and AUROC OvO. Does anyone know in what particular cases we would use AUROC OvR vs. AUROC OvO? In the general academic ...
2
votes
2answers
58 views

Is the PR AUC invariant under label flip?

The ROC-AUC curve is invariant under a flip of the labels. I don't know if its a famous result so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
1
vote
0answers
47 views

Orange ROC analysis widget [closed]

ROC analysis widget has an option to change prior target class probability. I'd like to know, when and how it should be used. Playing with it changes the slope of iso-performance line. Each classifier ...
1
vote
1answer
351 views

Micro Average vs Macro Average for Class Imbalance

I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional ...
2
votes
0answers
27 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
2
votes
0answers
286 views

AUC on ROC Curve near 1.0 for Multi-Class CNN but Precision/Recall are not perfect?

I am building a ROC Curve and calculating AUC for multi-class classification on the CIFAR-10 dataset using a CNN. My overall Accuracy is ~ 90% and my precision and recall are as follows: ...
0
votes
1answer
202 views

AUC ROC Curve multi class Classification

Here is the part of the code for ROC AUC Curve calculation for multiple classes. ...
1
vote
1answer
45 views

Algorithm for Binary classification

I have a data set with huge number of features ( Approximately 3000) and a binary target variable . The reason I have too many features is because of one hot encoding many categorical variables in ...
0
votes
0answers
27 views

Trying to plot REC curve and calculate AUC in R but getting error

I am working on regression machine learning techniques and calculating AUC (plotting REC) by using below paper link: http://homepages.rpi.edu/~bennek/papers/rec.pdf My Code; REC <- function(y_val , ...
5
votes
2answers
124 views

Confused AUC ROC score

I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a cross validation technique (to ...
1
vote
0answers
17 views

How to compute a confidence interval for AUC?

I found that in results of several binary classification problems, people report an AUC value together with a CI. I wonder how those CIs are computed. Is there a close-formed formula to compute them ...
1
vote
2answers
348 views

Don't understand why I get an inverse ROC curve for SVM (Python)

I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves ...
0
votes
0answers
18 views

confused AUC ROC score [duplicate]

I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a 10 fold cross-validation ...
0
votes
1answer
36 views

XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC)

My dataset is of shape – 5621*8 (binary classification) Label/target : Success (4324, 77 %) & Not success (1297, 23 %) (success and Not success were been ...
5
votes
2answers
3k views

AUC-ROC for Multi-Label Classification

Hey guys I'm currently reading about AUC-ROC and I have understood the binary case and I think that I understand the multi-classification case. Now I'm a bit confused on how to generalize it to the ...
0
votes
2answers
172 views

Getting lower performance metrics when using GridSearchCV

I have defined an XGBoost model and would like to tune some of its hyperparameters. I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training ...
1
vote
2answers
319 views

How to compute AUC in gridsearchSV (multiclass problem)

I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the ...
1
vote
1answer
174 views

Confusion matrix and ROC AUC curves are not in sync

I created a classification model with three target classes and created a confusion matrix to measure the accuracy, here is the matrix code ...
-1
votes
2answers
3k views

Plotting ROC & AUC for SVM algorithm

Towards , the end of my program, I have the following code. model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00001) model.fit(train_data) Output ...
0
votes
1answer
58 views

Confusion Matrix and AUC in univariate Anomaly Detection

In the code I upon a csv file which only has one column. The data in there in not that important just normal numbers. ...
2
votes
0answers
15 views

Connection between prob output LogisticReg/SVM and ROC

I have the following ROC generated using LPOCV and Logistic regression or SVM (l2 norm). Now, let's say I have a test set containing 10 patients and I get that the probabilities of those patients to ...
0
votes
1answer
176 views

Imbalanced dataset - Positive majority class

My dataset consists of 150 patients where 50 are controls/healthy (negative) and 100 are sick (positive). If I want my model to have high sensitivity at hight specificity, in other words to have low ...
2
votes
2answers
40 views

How much can the AUC improve comparing the raw dataset and the feature engineered dataset?

Let's say I put the following two datasets in the best possible model (same model for both): A raw dataset, the variables as they came just from the query. A feature-engineered dataset, with hundreds ...
2
votes
2answers
143 views

What is AUC - ROC Curve?

AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. Is Roc the ...
1
vote
1answer
13 views

Truncating float/doubles for reproducibility

I deploy machine learning models (typically GPU) to a variety of environments. I work sort of at the edge of ML R&D and devops, so I am really big into reproducibility, and one thing that drives ...
1
vote
0answers
269 views

How to create an roc plot and calculate AUC for an svm (that does not return probabilities)?

I have some SVM classifier outputting final classifications for every sample in the test set, something like 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1 and so on. The "...
1
vote
1answer
510 views

XGBOOST missing_value feature degrades my performance?

I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-...
1
vote
0answers
108 views

Regression for binary classification and AUC metric

In the kaggle forums I found an example model where someone was using XGBRegressor for a binary (0/1) classification problem (sorry, cannot find the link any more). This was for a competition where ...