I am about to implement an Autoencoder to detect anomalies. Therefore, e.g., in my test set, there is a situation where the data stream broke for some days. This results in a lack of data and should be flagged. I thought the AE would easily catch that but actually I figure that he doesn't as there is, quite obviously, no data to calculate the reconstruction error against.

Now I wonder if I miss the forest for the trees. I fit/predict with the AE through y=X and just using preds = AE.fit(X) and the reconstruction error by test_error = np.abs(y[:, 1] - preds[:, 1])

array([[-1.73103225,  1.6792525 ],
       [-1.72899455,  1.70540111],
       [-1.72695684,  1.51829725]])

The first column represents the time column (hourly data, transformed and standardscaled to numeric values), the second column holds current values. So, basically, the background is

current against time

Do I miss a crucial aspect in my concept? And/or is there a method how I can detect the situation, depicted below? I know that I can, in principle, detect it relatively easy just by checking whether there are consecutive time steps. But atm I try to catch as much situations as possible with one method. Afterwards I will add other/minor check routines. My native assumption was that I create a dataframe where all time steps/hours are given and add the real values to it, according to the correct time steps. The cells, where no real value is available (where the data stream was interrupted), I filled with zeros.

AE applied to modified data: modified-data As described, this doesn't help as I have no data to check against, so the AE just reconstructs these zeros as well.. but maybe I'm so caught in my thoughts that I hope some other oppinions could give me some hints.

AE applied to real data: Real data


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.