I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F parameters) which parameters (the 10/20/30 most impacting (A parameters)) to create a model for detecting anomalies.

Because I think that in these +600 parameters, all of them will not be useful and will disturb my model more than anything else.

I was thinking of using a heatmap / pearson to find the parameters that impact the most my F parameters.

It seems to me that my case is similar to a model with multivariate outputs, with for output my F parameters tell me if I'm wrong! So my questions are do you know how I can find which are the 10 most impacting A parameters for each B parameters and if the assumptions (heatmap/pearson) are good or if I have to do otherwise?

As I don't have the labels, I'm thinking of using unsupervised machine learning, and I'd like either an output telling me if it's OK / Not OK or an output with OK / NOT OK + which parameter triggers the NOT OK, which model do you advise me to use for this kind of problem? Mahalanobis Distance (MD) maybe ? I have no idea I am kinda new with those models

  • $\begingroup$ What is your definition of "anomaly" vs a normal instance? Without a labeled dataset, such a definition is critical. $\endgroup$
    – Jon Nordby
    Sep 22, 2022 at 19:28
  • $\begingroup$ The difference between A and F parameters are a bit unclear in your description. Are you saying that for each F parameters, there is some stable/predictable patterns with some of the A parameters - and it is these you wish to find? $\endgroup$
    – Jon Nordby
    Sep 22, 2022 at 19:31


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.