Dealing with informative missingness

Question

How can I deal with a time series that contains missing data which means something?

So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a loss of information.

So, you can think of it as something like "measure x only if condition y was met". In my case a certain price is measured only if there are people interested in that on a given day. This leads to a time series with several thousands data points, of which around 40% contains no value.

I want to use a 1D-CNN as part of an anomaly detector in order to find errors in the data. The CNN delivers predictions which are compared to the actual data. If the difference between the prediction and the real data point is above a certain threshold I want to mark that as an anomaly. (the missing_values however are no anomaly)

I thought of using embeddings to deal with missing values but am not sure if this would be the correct way of handling this.

Answer 1

I don't see how embeddings can help. Embeddings serve to lower the dimensionality of your data. I think you just need to introduce a single feature that is 1 if there were people interested and 0 if there were none. Then if you have sufficient data, the convnet should be able to understand the relation between the time-series and the feature you added.

It's a bit hard without having any idea about what your data looks like, but you might also be able to ignore the missing data, and train a model that predicts a continuous time series. Then the model may flag an anomaly when there is data missing, but you only actually declare an anomaly if the data is not missing and the anomaly is real.

Answer 2

I think this paper (Recurrent Neural Networks for Multivariate Time Series with Missing Values) tries to solve a similar problem of yours. Their idea is to train a neural network to capture the missing patterns and then combine that information to the main network to improve the performance.

There's a reimplementation here but I don't know about the quality of the source code.

Answer 3

Much later I found what I was actually looking for: Entity Embeddings

Basically I add contextual information to the neural network where I state if the field was missing is therefor zero.

I split up each field into 3 categories per field: zero, non-zero, missing, because there are multiple patterns in the data within the fields, that I am not always aware of.

I found this solution by reading this paper: Outlier Detection for Multidimensional Time Series using Deep Neural Networks (DOI:10.1109/MDM.2018.00029)