Evaluation metric for time-series anomaly detection

Question

I have a question for AI or data experts. I'm writing a paper

My dataset are time-series sensor data and anomaly ratio is between 5% and 6%

1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?

When empirically studying this issue, I found some papers use precision/recall/F1 and some papers use ROC-AUC .

Considering that positive samples(anomalies) are relatively less than negative samples(normal points), which one is better?

I'm confused with this issue

2. If I use precision/recall/F1 , should I check precision/recall/F1 only for positive class ?

I think because the number of positive samples are sparse, it's not appropriate to check precision/recall/F1 only for positive class

Thus, should I check precision/recall/F1 for both positive class and negative class?

If that's right, can I report precision/recall/F1 with macro avg in my paper?

(you can see the picture below. I used classification_report in sklearn library)

Thank you for your explanation !

Answer 1

Hi and welcome to the community!

1. Don't get confused between those. they are different ways of explaining the same concept. The point is that in such problems with very imbalanced class populations, you need to use an evaluation metric which considers the effect of fined detailed inspection i.e. TP, FP, TN and FN. Precision/Recall and AUC/ROC both use them.

But What's the main difference between them? AUC/ROC give you a wonderful visual representation (of course along with a number) and Precision/Recall give you more comprehensive detailed numerical evaluation. So the first is good for comparing several models and second one is better for deep inspection of each of models (of course they are still used vice-versa but "less nicer"). Do not even hesitate to include both. Just enriches the evaluation section of your paper.

2. Positive class is the main point of your paper however you also want to keep a track of performance on trivial class (normal points) so I suggest include both i.e. the classification_report that you posted is a great way of reporting results

3. You should certainly report Macro Average! In imbalanced problems you actually use precision/recall or auc/roc to get rid of something and weighted average is exactly calculation that thing!

That "thing" is affecting evaluation by size of big classes.

Example: here you see that precisions is very good for normal point and very bad for anomalies. What Weighted Average is telling you? it says 0.91 which is very good. But how is the performance? it is 0.1 on detecting anomalies which is the point of your paper! right? So be careful ... imbalanced problems should be evaluated by Macro Average.

Answer 2

F1 requires finding a threshold to decide which examples are normal and abnormal. This means that the f1 score is in some way dependent on how good you are in finding this threshold and not only on the model's anomaly score.

Your weighted f1 score is inflated highly by the good performance of normal data. Macro does not consider how many normal examples are in your test set. On the other side, you will see more normal data. Most people use weighted and write it in the text since the performance is better, and most people don't know what it means or do it the same way in their paper since it looks like very excellent results. Most of these approaches are never used in the real world. It is just an academic paper, and marketing is essential for grants! So it's usually fine to report the psychological better looking weighted.