I am working on an Anomaly Detection Problem and the algorithm I used is an Autoencoder Multivariate Gaussian. The problem with my data is that it is unlabeled and not correlated.

For example, let's say I want to predict anomaly in my system and I have features like:

vibration, temp, pressure, catalyst

Out of these features, only vibration is a good indicator of an anomaly, but other features (e.g. temp, pressure and cat) are independent of vibration. So even if there is an anomaly, the vibration feature will only show us some indication, but not the rest of the features.

I built the above-mentioned algorithm! Autoencoder gave me 99% accuracy during training time, but during testing, it totally failed (similar case with MVG).

So, do I need correlated features or do these algorithms should work on this data? Please suggest any other approach to solve this kind of problem.

  • $\begingroup$ Could you please clarify this: if only "vibration" is a good indicator of anomaly, why would you use other features to predict the anomaly? $\endgroup$ – Rodrigo Nader Jun 24 at 4:34
  • $\begingroup$ Vibration is good indicator but to build something generic model which will work across all systems , anyway if you know regarding my doubt asked above , please let me know $\endgroup$ – user3219871 Jun 24 at 5:36

To answer your question, no you don't necessarily need only correlated features in your model, most models should be able to ignore those that are not correlating. Although having only one correlating feature is not optimal. To build a general model, you will usually include any features that may be helpful in other use cases, even if they are not always used in each use case.

I think there are a few other points to work on with this model.

  1. Your model is over-fitting, need more information to troubleshoot this
  2. How is the train and test data split? What criteria was used?
  3. Does your test data look similar to the training data? Maybe you need more training cases.
  4. More features would be better. Are there any other features available?
  5. I am guessing low vibration is good and you are looking for high vibration values as the anomaly. If so, you can try to include only low vibration in your train data
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.