Hi I'm a CS graduate student
I have a question for AI or data experts. I'm writing a paper
My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6%
you can see the picture below. I used classification_report in sklearn library
But, I'm confused as to what value to report in my evaluation section ...
I think it is reasonable to report f1-score with macro avg (0.40)
Is it ok?
Thank you for your explanation !
Classic evaluation metrics are heavily affected by skewed data. In your case, since you have imbalanced data, you should definitely avoid those such as accuracy. For example,
Imagine you have a test data with 100 records. 3 of them have Class A and 97 have Class B. I create this model:
prediction = "B"
Simple as that, I always give the second class. I will get accuracy of 97%, but I fail to predict any of the records I really care.
This is important in cases like cancer prediction where early detection is important.
What you could use instead:
For an imbalanced dataset, you should report the base rate (5% anomaly, 95% regular observation), alongside your selected score; without that, the interpretation can lead to base-rate fallacy.
The common metrics derived from the confusion matrix will do. Be careful with ROC AUC, it tends to be over-optimistic compared to accuracy and is generally less understandable by practitioners. For highly imbalanced datasets, ROC AUC will be very close to 1 most of the time.
I personally like paired metrics, such as precision-recall, or specificity-sensitivity, which report separately type I and type II errors.
There are also metrics, which weigh these two errors unequally. F1-score focuses on the lower, e.g., precision 0.1, and recall 0.9 yields F1 0.18.
Check out the USENIX paper Do's and Don'ts of Machine Learning in Computer Security, which mentions metrics used for imbalanced datasets.
