1
$\begingroup$

I have a question for AI or data experts. I'm writing a paper

My dataset are time-series sensor data and anomaly ratio is between 5% and 6%

1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?

When empirically studying this issue, I found some papers use precision/recall/F1 and some papers use ROC-AUC .

Considering that positive samples(anomalies) are relatively less than negative samples(normal points), which one is better?

I'm confused with this issue

2. If I use precision/recall/F1 , should I check precision/recall/F1 only for positive class ?

I think because the number of positive samples are sparse, it's not appropriate to check precision/recall/F1 only for positive class

Thus, should I check precision/recall/F1 for both positive class and negative class?

If that's right, can I report precision/recall/F1 with macro avg in my paper?

(you can see the picture below. I used classification_report in sklearn library)

Thank you for your explanation !

enter image description here

$\endgroup$
0
$\begingroup$

Hi and welcome to the community!

  1. Don't get confused between those. they are different ways of explaining the same concept. The point is that in such problems with very imbalanced class populations, you need to use an evaluation metric which considers the effect of fined detailed inspection i.e. TP, FP, TN and FN. Precision/Recall and AUC/ROC both use them.

But What's the main difference between them? AUC/ROC give you a wonderful visual representation (of course along with a number) and Precision/Recall give you more comprehensive detailed numerical evaluation. So the first is good for comparing several models and second one is better for deep inspection of each of models (of course they are still used vice-versa but "less nicer"). Do not even hesitate to include both. Just enriches the evaluation section of your paper.

  1. Positive class is the main point of your paper however you also want to keep a track of performance on trivial class (normal points) so I suggest include both i.e. the classification_report that you posted is a great way of reporting results

  2. You should certainly report Macro Average! In imbalanced problems you actually use precision/recall or auc/roc to get rid of something and weighted average is exactly calculation that thing!

That "thing" is affecting evaluation by size of big classes.

Example: here you see that precisions is very good for normal point and very bad for anomalies. What Weighted Average is telling you? it says 0.91 which is very good. But how is the performance? it is 0.1 on detecting anomalies which is the point of your paper! right? So be careful ... imbalanced problems should be evaluated by Macro Average.

$\endgroup$
4
  • $\begingroup$ Thank you for your detailed explanation ! Um... according to your advice, Is it Ok that I report precision is 0.5265 and recall is 0.5033 ? $\endgroup$ Jun 23 at 8:13
  • $\begingroup$ And, "What Macro Average is telling you? it says 0.91 which is very good." This your sentence is incorrect. Macro Average is 0.5265 $\endgroup$ Jun 23 at 8:17
  • $\begingroup$ It is. I would personally include detailed number as well but it is totally your choice. And in general, this model is not performing well so it will be difficult to make it look better using evaluation metrics. Model discovers only 1% of anomalies and only 10% of what it detects as anomaly is really anomaly. So the issue is one step before. But regardless of that issue, you choice of metrics and using them is pretty right $\endgroup$ Jun 23 at 8:20
  • $\begingroup$ Your comment was right. Typo. I updated the answer $\endgroup$ Jun 23 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.