1
$\begingroup$

I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several unsupervised machine learning models for this task, such as Isolation Forest and K-Nearest Neighbors (KNN), I'm facing challenges in providing solid justifications for why the predicted values are classified as anomalies.

Since I don't have access to labeled data for validation, I find it challenging to explain the rationale behind the model's decisions. My stakeholders and domain experts require interpretability in the anomaly detection process to trust and act upon the model's output effectively.

Therefore, I'd like advice on the best approach to validating the predictions. I'm open to empirical approaches, as well as tactical advice.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Jul 25, 2023 at 15:00
  • 1
    $\begingroup$ To do a proper job at empirical evaluation or reasoning, you really need to get yourself some labeled data. Unsupervised learning does not mean unsupervised evaluation. Set up a label process somehow. $\endgroup$
    – Jon Nordby
    Commented Jul 25, 2023 at 15:47
  • $\begingroup$ This is not some crazy out-of-the-blue question. In my experience a lack of contact and ability to interview the stakeholders happens surprisingly often. Or the data is legacy and noone really understands it at depth so we have to decide upon the criteria ourselves. $\endgroup$
    – WestCoastProjects
    Commented Jul 31, 2023 at 18:44
  • $\begingroup$ Not at all crazy. The struggle is real. But at some point it is might no longer be a Data Science question though - but perhaps more about project management and organizational psychology... $\endgroup$
    – Jon Nordby
    Commented Jul 31, 2023 at 23:26

1 Answer 1

Reset to default
4
$\begingroup$

Your domain experts and other stakeholders are primary sources of information. Work them and work with them in order to find out what is an appropriate definition for "normal" and "abnormal". A key element to this will be interviews. It is useful to start with some open/unstructured. But then to move to asking them particularities of the data - typical cases. And then showing some data and asking them about how they would classify, what kind of process and reasoning they would apply.

The goal of such a process you should be able to extract:

  • A) Practical definitions of "normal" and "abnormal".
  • B) Examples of both normal and abnormal scenarios
  • C) A set of sound feature extraction strategies
  • D) A set of evaluation metric(s) that is appropriate, at least for relative ranking of performance

Which is sufficient to implement a system. It might be that the outcome is purely heuristics, or visualization/dashboard/reporting tools, or very simple unsupervised learning on top of problem-specific feature extraction.

And if done right - they will trust your model (and you) - because they have been involved every step of the way.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is great. If you can get it. What if you can't? $\endgroup$
    – WestCoastProjects
    Commented Jul 31, 2023 at 18:41
  • $\begingroup$ If you neither have labeled data nor access to skilled humans? A) Escalate the situation until you have what you need B) Inform that the project cannot be completed due to missing information C) Get creative and wing it D) Deliver something shitty/broken $\endgroup$
    – Jon Nordby
    Commented Jul 31, 2023 at 23:24
  • $\begingroup$ Yea Im interested in C which happens frequently $\endgroup$
    – WestCoastProjects
    Commented Aug 1, 2023 at 0:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.