I am about to implement an Autoencoder to detect anomalies. Therefore, e.g., in my test set, there is a situation where the data stream broke for some days. This results in a lack of data and should be flagged. I thought the AE would easily catch that but actually I figure that he doesn't as there is, quite obviously, no data to calculate the reconstruction error against.
Now I wonder if I miss the forest for the trees. I fit/predict with the AE through
y=X and just using
preds = AE.fit(X) and the reconstruction error by
test_error = np.abs(y[:, 1] - preds[:, 1])
X[0:3] array([[-1.73103225, 1.6792525 ], [-1.72899455, 1.70540111], [-1.72695684, 1.51829725]])
The first column represents the time column (hourly data, transformed and standardscaled to numeric values), the second column holds current values. So, basically, the background is
current against time
Do I miss a crucial aspect in my concept? And/or is there a method how I can detect the situation, depicted below? I know that I can, in principle, detect it relatively easy just by checking whether there are consecutive time steps. But atm I try to catch as much situations as possible with one method. Afterwards I will add other/minor check routines. My native assumption was that I create a dataframe where all time steps/hours are given and add the real values to it, according to the correct time steps. The cells, where no real value is available (where the data stream was interrupted), I filled with zeros.
AE applied to modified data: As described, this doesn't help as I have no data to check against, so the AE just reconstructs these zeros as well.. but maybe I'm so caught in my thoughts that I hope some other oppinions could give me some hints.