1
$\begingroup$

I have a dataset with 5 features corresponding to 5 sensors that measure each three seconds the state of an accelerator. It is structured as well:

Sensor 1 | Sensor 2 | Sensor 3 | Sensor 4 | Sensor 5 | Label
   1.5       1.1        0.8        1.2        1.2       0
   1.2       1.4        1.4        1.4        1.1       0
   1.2       1.1        1.2        1.3        1.5       0

The label indicates if the time series is anomaly(=1) or not(=0). I have an anomaly detection task, and the frameworks I've chosen (1, 2) give me as output an array with length 3 where I have the labels predicted: (0, 1, 0). I usually worked with anomaly detection frameworks which gave me a threshold and I could have easily marked the values above it as anomalies.

In this specific case, with this array of length 3, is it right to assume that I could rewrite the following dataset as this? (True = Anomaly, False = normal)

Sensor 1 | Sensor 2 | Sensor 3 | Sensor 4 | Sensor 5 |
 False       False      False     False       False    
 True        True       True      True        True    
 False       False      False     False       False     

So, instead mark one value at time, it directly mark all the time series as anomaly?

$\endgroup$
7
  • $\begingroup$ Your labels are all 0 for the shown data. So why would any of the features be considered anomalous? $\endgroup$
    – Jon Nordby
    Oct 29, 2021 at 21:02
  • $\begingroup$ That’s an example where the algorithm doesn’t perform well and find an anomaly. What I’m interested to know is if the full row is considered anomalous or not $\endgroup$
    – Fabio
    Oct 29, 2021 at 22:38
  • $\begingroup$ Ok, so it is the output of the AD algorithms, not the ground truth. It would normally be called a prediction, to separate from ground truth. $\endgroup$
    – Jon Nordby
    Oct 31, 2021 at 14:36
  • $\begingroup$ But for tips on how to better detect anomalies in your dataset, you will need to provide more info on your normal and abnormal data. What kind of data are considered to be anomalies? $\endgroup$
    – Jon Nordby
    Oct 31, 2021 at 14:38
  • $\begingroup$ @JonNordby the output I get are the labels, which identify where my anomalies are. In this case, since I got 1 on the second row, would it mean that all the values in that row are anomaly? $\endgroup$
    – Fabio
    Oct 31, 2021 at 16:39

1 Answer 1

2
+100
$\begingroup$

I believe the assumption you've made is incorrect (the whole row being anomalous or not).

To explain this thoroughly, you would need to know which algorithm you're using in order to detect whether or not the final label is 0 or 1.

For anomaly detection, you can approach the problem as a Supervised (pretty much classification problem), Unsupervised or Semi-Supervised. Assuming from your data, you've chosen the unsupervised approach.

Unsupervised Anomaly Detection

An unsupervised anomaly detection has a plethora of algorithm subsets; Distance Based, Statistical, Classification, Angle Based, DBscan, Neural Networks and more. Under those subsets lie algorithms that can help detect anomalous values such as unique NN architectures etc.

Some algorithms (most of the ones I've used) have some form of dimensionality reduction such as PCA. Due to that fact, its a lot more difficult to grasp whether a specific column (e.g Sensor 1 is anomalous on its own)

A better way to wrap your head around how/why a datapoint is tagged anomalous or not would be to plot a t-SNE graph.

If you're interested in me editing my answer to create an anomaly detection model that tags data points as anomaly or not (along with the anomaly score for you to be able to set your personal threshold) and plotting a t-SNE graph, let me know.

$\endgroup$
1
  • $\begingroup$ For my task of anomaly detection I need to use a deep learning approach, in particularly the two algorithms I linked in my question. The both give me as output a vector with length=number of samples per features, instead a matrix. I don't understand why and if my interpretation is correct $\endgroup$
    – Fabio
    Nov 1, 2021 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.