Unsupervised feature reduction for anomaly detection with autoencoders

Question

I am collecting a big number of generated numeric features for the task of unsupervised anomaly detection.

I can assume that all training data is considered normal.

I expect some of the generated features to be characterised with low standard deviation, for example, some features might be always 0 in the train examples. In contrast, I expect that some of these features will deviate in anomaly instances.

As I have a lot of feature, I strive to perform feature reduction / selection. However, using simple feature selection methods, will completely remove the non deviating features, affecting the upcoming detection for the worse.

I was thinking about using stacked auto-encoders for the sake of feature reduction, so that whenever a feature deviates a lot from the stdv it will affect all resulted features - causing a noticeable anomaly.

Will this technique work? if not, why? and what other technique could work for that.

Also, if it does, and I am planning of using deep auto-encoders for the sake of anomaly detection as well, is the first step of feature reduction redundent?

Answer 1

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features.

When outliers show up these values are different and the network was unable to construct these particular data points back and the error scores for these values should be typically higher.

I used this with Deeplearning4j with SparkDl4jMultiLayer with KLDivergence loss functions on all layers except output layer with MSE loss functions. with Sigmoid activation function.