Hot answers tagged

3

Related concepts, but not the same. ROC-receiver operating curve AUC area under the curve Thank this post for explanation : Abbreviations AUC = Area Under the Curve. AUROC = Area Under the Receiver Operating Characteristic curve. AUC is used most of the time to mean AUROC, which is a bad practice since as Marc Claesen pointed out AUC is ambiguous (...


3

Yes, the performance can vary a lot using feature engineering. Example: suppose a dataset where the response variable $y$ is true if $x$ is odd. x y 346 F 13 T 178 F 64 F 987 T ... Most learning models will fail to identify the pattern and will perform poorly, usually falling back to always predicting the majority class. However simply adding ...


2

For any classification problem if AUC<0.5, you are not performing better than random(0.5). Reason could be: Your classifier is over-fitted on the training set and performs very poorly on the test set. Your test sample might be very small. Your classifier is giving you the probability that the class is -1. Thus, you get a prediction (close to) 0 for a ...


2

I would say that the best possible model for the raw data would derive all the meaningful features that you would have created from the data anyway. And I would say that the best possible model for the feature-engineered model will remove/ignore unnecessary features. The best possible model would have AUC of 1 anyway. It makes all predictions correctly. ...


2

I'm not expert in this but as far as I know the proper way to test for equality modulo floating point imprecision is to compare the differences of the two values, i.e. instead of: trunc(a) == trunc(b) one would do: abs(a-b) <= epsilon where epsilon is the constant which represents an acceptable difference, e.g. $10^{-6}$. Of course this requires to ...


2

It's difficult to answer precisely without knowing the data and the task. Assuming it's a single column of values with no order involved, it boils down to finding the optimal threshold to separate regular values vs. anomalies. Given that you know which ones are the anomalies (labelled data), the problem can be treated as a binary classification task: a ...


2

df = pd.get_dummies(pred1) df.insert(loc=2,column='2',value=0) #print(df) add this before the for loop and instead of using pd.get_dummies(y_test) use only df


2

AUC can be defined as $P(X_1 > X_0)$ where $X_1$ is the score of a randomly chosen positive instance and $X_0$ is the score of a randomly chosen negative instance. Like in Simpson's "paradox", what you see could happen because the group has a relatively large effect on your target. For example, one group could be mostly positive, another could ...


2

I propose to look at this a bit differently. The different classification status of the two classes are related as follows: $TP_1=TN_0$ $FP_1=FN_0$ $FN_1=FP_0$ $TN_1=TP_0$ From there we have: $$FPR_0=\frac{FP_0}{FP_0+TN_0}=\frac{FN_1}{FN_1+TP_1}=1-\frac{TP_1}{FN_1+TP_1}=1-TPR_1$$ and naturally $FPR_1=1-TPR_0$ for the same reasons. Therefore when one ...


1

The question is actually about understanding what it means to "take imbalance into account": Micro-average "takes imbalance into account" in the sense that the resulting performance is based on the proportion of every class, i.e. the performance of a large class has more impact on the result than of a small class. Macro-average "...


1

You're assuming that the points are equally spaced along the fpr axis, which is generally not true. See e.g. the "Uniform grid" vs "Nonuniform grid" sections of the wikipedia article. You need something like delta_xs = np.diff(fpr) left_endpoints_y = tpr[:-1] right_endpoints_y = tpr[1:] trap_areas = 0.5 * (left_endpoints_y + ...


1

The ROC-AUC curves are used to find the best threshold that optimizes True Positive Rate vs False Positive Rate. Using it in a K-Fold cross-validation is a good practice to determine the best threshold to use. Then, your final test is here to validate that you did not overfit on some hyperparameters, including this threshold. So ROC-AUC must not be used ...


1

I made an experiment with a toy dataset in which first I trained a classifier with the original labels and plot the roc curve, then I switched the labels and train the same model on with the "new target" and even though the score in test set were the same, the roc auc plot looks slightly different. import numpy as np import matplotlib.pyplot as plt ...


1

First thing that comes to my mind is to do different encodings. There are some ways to deal with high cardinality categorical data such as: Label Encoding or the famous target encoding. Before anything else I will recommend changing the encoding type. But, since your question about which predictor use with small and space data. I will go still with logistic ...


1

One potential fix is to remove max_iter = 12 (which would set it to the scikit learn default of max_iter=-1). Using such a low value can lead to bad scores as you can see from the following example: from sklearn.model_selection import train_test_split from sklearn.svm import SVC from sklearn.metrics import plot_roc_curve from sklearn.datasets import ...


1

I am not sure if I understand your thinking, but my understanding is following - maybe it can help you see it from another perspective. The multi-label classification problem with n possible classes can be seen as n binary classifiers. If so, we can simply calculate AUC ROC for each binary classifier and average it. This is a bit tricky - there are ...


1

There are two things that might help you: 1) If you use a classifier $f$ that returns a value $f(x)\in[0,1]$ between 0 and 1 instead of a direct class assignment, then you can use a threshold $\theta\in[0,1]$: $f(x)=\begin{cases} 1,& \text{if } f(x)\geq\theta\\ 0, & \text{else } \end{cases}$ The higher the value of $\theta$, the more you ...


1

Oh, I think I've finally got it. It's just an averaging problem: for each fold in your k-fold cross-validation, you get perfect auROC, but at the default threshold of 0.5 your hard classifiers (for each fold) sometimes have $FPR=0$ and $TPR<1$, but some other times $FPR>0$ and $TPR=1$. Then averaging you are able to get both $\operatorname{mean}(FPR)&...


1

With Success already being the larger class, you probably shouldn't be using a scale_pos_weight larger than one: you want to scale the positive class's contribution to the loss function down rather than up. I suspect that's what's happening in the first case. With scale_pos_weight=75, the model ends up basically only caring about the positive class, ...


1

Imbalanced class means the count of one class is too low compared to the count of other Class. This means Model will have little opportunity to learn the minority Class. We have these option to handle the issue. Key goal is to reduce the fog created by the majority Class and let the Model see the Minority class too - Weighted Class - This instructs the ...


1

your resume is quite good, but I'm not comfortable in dividing the broad discussion to those three more or less sharply separated roads. But indeed, often a technique similar to one of those is chosen. Just let me underline something about them: the Area Under the Curve (AUC) of Precision and Recall has been shown as being slightly better than AUC ROC, but ...


1

Your confusion seems to stem from this line: print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100)) The best_score_ is not exactly a training score (nor is it an unbiased estimate of future performance*): as you say, it's the averaged score across different fold splits, but each of the scores that get averaged are the performance of the ...


1

Metrics are independent from ML algorithms, so it doesn't matter which algorithms did you use. To calculate multiclass AUC you could use lib pRoc in R or use code this link(in Python). Sources: https://web.expasy.org/pROC/ https://medium.com/@plog397/auc-roc-curve-scoring-function-for-multi-class-classification-9822871a6659


1

If you are performing a binary classification task then the following code might help you. from sklearn.model_selection import GridSearchCV for hyper-parameter tuning. from sklearn.linear_model import SGDClassifier by default, it fits a linear support vector machine (SVM) from sklearn.metrics import roc_curve, auc The function roc_curve computes the ...


Only top voted, non community-wiki answers of a minimum length are eligible