Skip to main content
Share Your Experience: Take the 2024 Developer Survey
487 votes

Micro Average vs Macro average Performance in a Multiclass classification setting

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and ...
pythiest's user avatar
  • 4,979
67 votes
Accepted

When is precision more important over recall?

For rare cancer data modeling, anything that doesn't account for false-negatives is a crime. Recall is a better measure than precision. For YouTube recommendations, false-negatives is less of a ...
SmallChess's user avatar
  • 3,560
60 votes
Accepted

What is the difference between bootstrapping and cross-validation?

Both cross validation and bootstrapping are resampling methods. bootstrap resamples with replacement (and usually produces new "surrogate" data sets with the same number of cases as the original ...
cbeleites unhappy with SX's user avatar
46 votes

Micro Average vs Macro average Performance in a Multiclass classification setting

This is the Original Post. In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the ...
Rahul Reddy Vemireddy's user avatar
28 votes

Micro Average vs Macro average Performance in a Multiclass classification setting

In a multi-class setting micro-averaged precision and recall are always the same. $$ P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\ R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c} $$ where c ...
David Makovoz's user avatar
28 votes
Accepted

Train/Test Split after performing SMOTE

When you use any sampling technique (specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you do the training, you use the test set (...
Bashar Haddad's user avatar
23 votes
Accepted

How many features to sample using Random Forests

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following: The number of randomly selected features can influence the generalization error in two ways: ...
oW_'s user avatar
  • 6,377
22 votes

How to define a custom performance metric in Keras?

You have to use Keras backend functions. Unfortunately they do not support the &-operator, so that you have to build a workaround: We generate matrices of the ...
pexmar's user avatar
  • 381
21 votes

When is precision more important over recall?

I can give you my real case when recall is more important: We have thousands of free customers registering in our website every week. The call center team wants to call them all, but it is imposible, ...
miguelbadajoz's user avatar
21 votes
Accepted

What are the disadvantages of accuracy?

A common complaint about accuracy is that it fails when the classes are imbalanced. For instance, if you get an accuracy of $98\%$, that sounds like a high $\text{A}$ in school, so you might be pretty ...
Dave's user avatar
  • 3,903
18 votes
Accepted

Macro- or micro-average for imbalanced class problems

The choice of a metric depends on how you rank the importance of your classes and what you value from a classifier. Let's look at your example: For example if we have a data set with 90%-10% class ...
Djib2011's user avatar
  • 7,998
16 votes
Accepted

Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

The difference between macro and micro averaging for performance metrics (such as the F1-score) is that macro weighs each class equally whereas micro weights each sample equally. If the distribution ...
Alec K's user avatar
  • 311
12 votes
Accepted

Irregular Precision-Recall Curve

This is definitely possible. When you are reducing the threshold, you will never decrease the recall (you can only flag more of the positive examples as positive). Precision is looking at all the ...
Jan van der Vegt's user avatar
11 votes

Neural Networks - Loss and Accuracy correlation

Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due ...
Jan van der Vegt's user avatar
11 votes

When is precision more important over recall?

Which is more important simply depends on what the costs of each error are. Precision tends to involve direct costs; the more false positives you have, the more cost per true positive you have. If ...
Acccumulation's user avatar
11 votes
Accepted

Is the F1 Score sensitive to the threshold?

Yes, it is. The formula: $$F_1 = \frac{2TP}{2TP + FP + FN}$$ There is no symmetry. You have only $TP$ in the nominator which means that $F_1=0$ if $threshold=1$.
Simon Larsson's user avatar
11 votes
Accepted

What is Continuous Ranked Probability Score (CRPS)?

CRPS is in a sense just the mean square error (MSE) of your predicted cumulative density function (CDF) and the true CDF. The CRPS generalizes the MAE (Mean Absolute Error) to the case of ...
Pluviophile's user avatar
  • 3,898
10 votes

When is precision more important over recall?

Although in some situations recall may be more important than precision (or vice versa), you need both to get a more interpretable assessment. For instance, as noted by @SmallChess, in the medical ...
Ben's user avatar
  • 2,572
10 votes

When is precision more important over recall?

I had a tough time remembering the difference between precision and recall, until I came up with this mnemonic for myself: PREcision is to PREgnancy tests as reCALL is to CALL center. With a ...
Jennifer Darrouzet's user avatar
9 votes
Accepted

Difference between learning_curve and validation_curve

Both curves show the training and validation scores of an estimator on the y-axis. A learning curve plots the score over varying numbers of training samples, while a validation curve plots the score ...
MB-F's user avatar
  • 286
9 votes

Do I need validation data if my train and test accuracy/loss is consistent?

You don't always need 3 separate datasets. You usually split a dataset into 3 if you are doing some parameter or hyperparameter tuning before choosing a final model. Tuning will usually add bias from ...
Donald S's user avatar
  • 1,959
8 votes
Accepted

Micro Average vs Macro Average for Class Imbalance

The question is actually about understanding what it means to "take imbalance into account": Micro-average "takes imbalance into account" in the sense that the resulting ...
Erwan's user avatar
  • 25.5k
8 votes

What are the disadvantages of accuracy?

In general, the main disadvantage of accuracy is that it masks the issue of class imbalance. For example if the data contains only 10% of positive instances, a majority baseline classifier which ...
Erwan's user avatar
  • 25.5k
7 votes

Micro Average vs Macro average Performance in a Multiclass classification setting

Assume that we are classifying an email into one of the three groups: urgent, normal and spam. We compare the predicts with the ground truth labels, then we get the following confusion matrix and the ...
Lerner Zhang's user avatar
7 votes

What is the difference between bootstrapping and cross-validation?

Bootstrapping is any test or metric that relies on random sampling with replacement.It is a method that helps in many situations like validation of a predictive model performance, ensemble methods, ...
Christos Karatsalos's user avatar
7 votes

precision@k and recall@k

A quick answer. Note: Checking the references I could access fully, there are no discrepancies between the definitions as long as the terms are translated correctly. Some definitions: Relevant items: ...
Nikos M.'s user avatar
  • 2,353
6 votes
Accepted

XGBoost increase the error when changing evaluation function

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and <...
tlorieul's user avatar
  • 1,009
6 votes

Micro Average vs Macro average Performance in a Multiclass classification setting

That's how it should be. I had the same result for my research. It seemed weird at first. But precision and recall should be the same while micro-averaging the result of multi-class single-label ...
Saghan Mudbhari's user avatar
6 votes
Accepted

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by ...
oW_'s user avatar
  • 6,377
6 votes

When is precision more important over recall?

Accumulation has a great answer on how you can come up with more examples explaining the importance of precision over recall and vice versa. Most of the other answers make a compelling case for the ...
Tanmay's user avatar
  • 61

Only top scored, non community-wiki answers of a minimum length are eligible