2
$\begingroup$

I have a question for AI or data experts. I'm writing a paper

My dataset are time-series sensor data and anomaly ratio is between 5% and 6%

1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?

When empirically studying this issue, I found some papers use precision/recall/F1 and some papers use ROC-AUC .

Considering that positive samples(anomalies) are relatively less than negative samples(normal points), which one is better?

I'm confused with this issue

2. If I use precision/recall/F1 , should I check precision/recall/F1 only for positive class ?

I think because the number of positive samples are sparse, it's not appropriate to check precision/recall/F1 only for positive class

Thus, should I check precision/recall/F1 for both positive class and negative class?

If that's right, can I report precision/recall/F1 with macro avg in my paper?

(you can see the picture below. I used classification_report in sklearn library)

Thank you for your explanation !

enter image description here

$\endgroup$

2 Answers 2

-1
$\begingroup$

Hi and welcome to the community!

  1. Don't get confused between those. they are different ways of explaining the same concept. The point is that in such problems with very imbalanced class populations, you need to use an evaluation metric which considers the effect of fined detailed inspection i.e. TP, FP, TN and FN. Precision/Recall and AUC/ROC both use them.

But What's the main difference between them? AUC/ROC give you a wonderful visual representation (of course along with a number) and Precision/Recall give you more comprehensive detailed numerical evaluation. So the first is good for comparing several models and second one is better for deep inspection of each of models (of course they are still used vice-versa but "less nicer"). Do not even hesitate to include both. Just enriches the evaluation section of your paper.

  1. Positive class is the main point of your paper however you also want to keep a track of performance on trivial class (normal points) so I suggest include both i.e. the classification_report that you posted is a great way of reporting results

  2. You should certainly report Macro Average! In imbalanced problems you actually use precision/recall or auc/roc to get rid of something and weighted average is exactly calculation that thing!

That "thing" is affecting evaluation by size of big classes.

Example: here you see that precisions is very good for normal point and very bad for anomalies. What Weighted Average is telling you? it says 0.91 which is very good. But how is the performance? it is 0.1 on detecting anomalies which is the point of your paper! right? So be careful ... imbalanced problems should be evaluated by Macro Average.

$\endgroup$
4
  • $\begingroup$ Thank you for your detailed explanation ! Um... according to your advice, Is it Ok that I report precision is 0.5265 and recall is 0.5033 ? $\endgroup$ Jun 23, 2021 at 8:13
  • $\begingroup$ And, "What Macro Average is telling you? it says 0.91 which is very good." This your sentence is incorrect. Macro Average is 0.5265 $\endgroup$ Jun 23, 2021 at 8:17
  • $\begingroup$ It is. I would personally include detailed number as well but it is totally your choice. And in general, this model is not performing well so it will be difficult to make it look better using evaluation metrics. Model discovers only 1% of anomalies and only 10% of what it detects as anomaly is really anomaly. So the issue is one step before. But regardless of that issue, you choice of metrics and using them is pretty right $\endgroup$ Jun 23, 2021 at 8:20
  • $\begingroup$ Your comment was right. Typo. I updated the answer $\endgroup$ Jun 23, 2021 at 8:21
0
$\begingroup$

F1 requires finding a threshold to decide which examples are normal and abnormal. This means that the f1 score is in some way dependent on how good you are in finding this threshold and not only on the model's anomaly score.

Your weighted f1 score is inflated highly by the good performance of normal data. Macro does not consider how many normal examples are in your test set. On the other side, you will see more normal data. Most people use weighted and write it in the text since the performance is better, and most people don't know what it means or do it the same way in their paper since it looks like very excellent results. Most of these approaches are never used in the real world. It is just an academic paper, and marketing is essential for grants! So it's usually fine to report the psychological better looking weighted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.