1
$\begingroup$

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't have the labels, I would like to know if there is a way to do cross validation in that case?

Any help is much appreciated

Improve this question
$\endgroup$
5
  • $\begingroup$ Hi - if you don't have any labels then you'll be restricted to unsupervised methods (so cross-validation is going to be difficult). What sorts of thoughts did you have for evaluating your solution so far? Do you have an idea of how many anomalies your dataset contains and how they differ from the norm? $\endgroup$
    – redhqs
    Commented Aug 25, 2018 at 9:46
  • $\begingroup$ Hey! since it is warranty data, only people with business knowledge can evaluate the results. That is what I am doing right now. And I have no idea how many anomalies I will have. $\endgroup$
    – Ashwini
    Commented Aug 25, 2018 at 10:03
  • $\begingroup$ Sounds challenging! I think you could use an autoencoder to build a representation of your warranties, then find outliers by evaluating which points in your data are the furthest from that representation by some sort of distance measure. You could rank by furthest distance and evaluate them with your domain experts to see if the outliers are true anomalies or if you need to reconsider the features you're passing to your autoencoder. $\endgroup$
    – redhqs
    Commented Aug 25, 2018 at 10:51
  • $\begingroup$ Ya that is what I am doing right now :) Thank you for your suggestion. Is there any other way to validate? $\endgroup$
    – Ashwini
    Commented Aug 25, 2018 at 13:11
  • $\begingroup$ Unless I'm missing something, I don't think you're going to be able to validate until you get some feedback or labels from your domain experts! $\endgroup$
    – redhqs
    Commented Aug 26, 2018 at 13:03

3 Answers 3

Reset to default
0
$\begingroup$

We had the same issue. It's a big problem in industry. We identified abnormalities and struggles a lot to validate. As this is unsupervised techniques. ( would have been easier if it was supervised -https://machinelearningstories.blogspot.com/2018/07/anomaly-detection-anomaly-detection-by.html) After few months we took this approach-

1) ask warranty data expert to verify results.
2) If possible can they lebel entire data, so that problem becomes supervised?
3) remove false positive from unsupervised results as part of iterative improvement of model.

I can share more information, if you are interested. We struggled a lot for validation and faced a lot of friction. hope you will avoid those.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Hello Arpit. Thank you for the answer. I followed the exact same steps 1 and 3. Step 2 is not possible because of huge size. More information would be really helpful. $\endgroup$
    – Ashwini
    Commented Oct 2, 2018 at 17:56
  • $\begingroup$ 1) We had few known failures, so we had found large number of abnormalities before failure events. 2) checked all the high abnormalities data points ( esp. continuous high abnormality) and looked in actual data if we can related them. $\endgroup$
    – Arpit Sisodia
    Commented Oct 3, 2018 at 9:23
  • $\begingroup$ your assumption is abnormality= 'relationship( linear/non linear) between variables have changes. is it acceptable definition of abnormality in your domain? if yes, error of auto encoder reflects abnormality strength. $\endgroup$
    – Arpit Sisodia
    Commented Oct 3, 2018 at 9:25
  • $\begingroup$ Yes you are right. The abnormality is based on the relationship between variables. So in the context of warranty data, the anomalies are claims which deviates from the normal patterns. I have one more question. When you ask warranty data experts to verify results, how can we rank them and show only few anomalies? because I guess verifying 1000 anomalies is not practically possible $\endgroup$
    – Ashwini
    Commented Oct 3, 2018 at 12:02
  • $\begingroup$ 1) error will be higher for abnormal data. Error can be taken as abnormality indicator, so can rank them . 2) what makes more sense- point abnormality or continious set of abnormalities. In our scenario, point abnormality does't useful so we look for continuous abnormalities. $\endgroup$
    – Arpit Sisodia
    Commented Oct 4, 2018 at 10:58
1
$\begingroup$

Autoencoders are used to reduce the dimensionality of the feature space. They can capture nonlinearities that other dimensionality reduction teqniques like PCA can not. Autoencoders are build by training the model to reproduce the input. In this case you can split the data set into three:

  • training
  • cross validation
  • testing

Train you model using the training set, check the performance by looking at your loss for the cross validation set, make some changes, and repeat. After you have a model you are confident in, then using the test data set as the final word in its performance.

I think the main point is that your loss should have something to do with the difference between an element of the data set, and the encoded-then-decoded version of that element. Therefore you cross validate against a separate chunk of data as you tune the hyper parameters of the encoder.

Improve this answer
$\endgroup$
0
$\begingroup$

No, cross validation requires labelled data since you are measuring performance of prediction against ground truth (the labels).

If in your case, since this is part of business operations, you can ask for feedback on which observations were correctly classified to generate a set of labels. You could then use a nearest neighbours type algorithm to identify other similar cases.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.