201

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average ...


32

For rare cancer data modeling, anything that doesn't account for false-negatives is a crime. Recall is a better measure than precision. For YouTube recommendations, false-negatives is less of a concern. Precision is better here.


25

Original Post - http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics. Tricky, but I found this very interesting. There are two methods by which you can ...


17

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following: The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees ...


17

Both cross validation and bootstrapping are resampling methods. bootstrap resamples with replacement (and usually produces new "surrogate" data sets with the same number of cases as the original data set). Due to the drawing with replacement, a bootstrapped data set may contain multiple instances of the same original cases, and may completely omit other ...


16

You have to use Keras backend functions. Unfortunately they do not support the &-operator, so that you have to build a workaround: We generate matrices of the dimension batch_size x 3, where (e.g. for true positive) the first column is the ground truth vector, the second the actual prediction and the third is kind of a label-helper column, that contains ...


14

In a multi-class setting micro-averaged precision and recall are always the same. $$ P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\ R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c} $$ where c is the class label. Since in a multi-class setting you count all false instances it turns out that $$ \sum_c FP_c = \sum_c FN_c $$ Hence P = R. In other words, ...


13

Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (ROC-AUC) and the area under the curve of a Precision-Recall curve (PR-AUC) lies in its tractability for unbalanced classes. They are very similar and have been shown to contain essentially the same information, however PR curves ...


12

When use any sampling technique ( specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you train you use the testing set ( which contains only original samples) to evaluate. The risk if you use your strategy is to have the original sample in training ( testing) and the synthetic sample ( that ...


12

I can give you my real case when recall is more important: We have thousands of free customers registering in our website every week. The call center team wants to call them all, but it is imposible, so they ask me to select those with good chances to be a buyer (with high temperature is how we refer to them). We don't care to call a guy that is not going ...


10

Yes, it is. The formula: $$F_1 = \frac{2TP}{2TP + FP + FN}$$ There is no symmetry. You have only $TP$ in the nominator which means that $F_1=0$ if $threshold=1$.


9

Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due to the backpropagation algorithm that requires the loss function to be differentiable. When your preferred loss is not directly optimizable (like the accuracy) ...


8

nDCG is used to evaluate a golden ranked list (typically human judged) against your output ranked list. The more is the correlation between the two ranked lists, i.e. the more similar are the ranks of the relevant items in the two lists, the closer is the value of nDCG to 1. RMSE (Root Mean Squared Error) is typically used to evaluate regression problems ...


7

Although in some situations recall may be more important than precision (or vice versa), you need both to get a more interpretable assessment. For instance, as noted by @SmallChess, in the medical community, a false negative is usually more disastrous than a false positive for preliminary diagnoses. Therefore, one might consider recall to be a more ...


7

Which is more important simply depends on what the costs of each error is. Precision tends to involve direct costs; the more false positives you have, the more cost per true positive you have. If your costs are low, then precision doesn't matter as much. For instance, if you have 1M email addresses, and it will cost $10 to send an email to all of them, it'...


6

Loss is more general than accuracy. In classification, you can go to 100% accuracy, where all the labels are predicted correctly. But what about regression or forecasting? There is no definition of 0% and 100% Loss can be optimized with various methods. In Numerical Methods class, you've learned to solve a function by optimizing it (which is minimizing $|y_{...


6

The choice of a metric depends on how you rank the importance of your classes and what you value from a classifier. Let's look at your example: For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% accuracy by assigning the majority class label. One minor correction is that this way you can achieve a ...


6

The difference between macro and micro averaging for performance metrics (such as the F1-score) is that macro weighs each class equally whereas micro weights each sample equally. If the distribution of classes is symmetrical (i.e. you have an equal number of samples for each class), then macro and micro will result in the same score. As an example for your ...


5

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and rmse as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting: dtrain = DMatrix(X_train, label=np.log1p(y_train)) where np.log1p(x) is equal ...


5

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values correlate with a good or bad performance of the ...


5

Both curves show the training and validation scores of an estimator on the y-axis. A learning curve plots the score over varying numbers of training samples, while a validation curve plots the score over a varying hyper parameter. The learning curve is a tool for finding out if an estimator would benefit from more data, or if the model is too simple (...


4

In order to improve your classifier, you have few options. Ensembling - make a group of classifier and let them predict together. Stacking, blending, bagging, boosting. Choice is yours. Hyper parameter tunning - you did not mention your tool, but i suppose that in every solid one is option to search in parameter space to find the best combination Sampling - ...


4

"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since it comes down to what is possible.


4

For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is simple convert the ratings into binary likes and dislikes based on some threshold. Essentially you would take the "likes" of a user. Find the user is the testing ...


4

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by class imbalance through the false positive rate FP/(FP+TN). If the number of negatives is a lot larger, this could be a potential issue. In short, the Gini Coefficient has similar pros and cons as ...


4

I think you are correct. The line should be loss_batch, logits_batch = sess.run([loss, logits], feed_dict={X: X_batch, Y:Y_batch})


4

Useful metrics in such scenario are: F1 Score (and precision / recall) ROC Curves (Metric is : Area Under the ROC Curve (AUC)) Few articles on how to choose metrics for a specific project are: Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista, What metrics should be used for evaluating a model on an imbalanced data set? by ...


4

It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure). But very often the goal ...


4

Just to clarify (and I think you've got this right, but I'm just being careful), it is best practice to: 1: Split your data into train and test 2: Split train into train and eval 3: Grid search over hyperparameters, for each combination, train on train, evaluate on eval. Select the hyperparameters which allow you to get the best score on the eval set ...


3

First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold. However, you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the ...


Only top voted, non community-wiki answers of a minimum length are eligible