2
$\begingroup$

I’m new into Machine Learning so here I am asking for a sanity check, if the question I am asking is even reasonable.

I have a Dataset of columns, so I want to call one of the columns from the csv using pandas.

Take one of the number from that column of numbers, do some unsupervised learning to determine if this value is an anomaly in this column and it belongs there or not.

The graph below shows, how I would see this process going. I am not sure on what I would do regarding what unsupervised method would be best for this case.

enter image description here

$\endgroup$
0
$\begingroup$

A few observations/questions/considerations:

  1. Wikipedia: Unsupervised learning is a type of self-organized Hebbian learning that helps find previously unknown patterns in data set without pre-existing labels.
  2. Being an anomaly is a label.
  3. Why not just Z-score it? https://en.wikipedia.org/wiki/Standard_score#Z-test . Calculate distribution: $\mu$ and $\sigma$, and check if value is within $\mu$ +/- 1.96 $\sigma$, $95\%$ chance it is from the normal distribution of the numbers in the column.
  4. Can the Value column be something else than numbers? (E.g. color names)
  5. Or could it be a time series with drift?

Edit:

Under the hood, Machine Learning is basically a set of smart statistics and decisions on subsets of columns and rows. But with only one column and no time series, there is nothing smart to subset or decide. Then essentially, it is an exceptional Z-score for the elements. You have to decide on how many $\sigma$ you want.

$\endgroup$
  • $\begingroup$ By the way, 432 is not an anomaly with the Z score method, the $\sigma$ is pretty high. $\endgroup$ – Pieter21 Jan 23 at 16:13
  • $\begingroup$ the 432 I know its not an anomaly that was just an example. bad one in this case. What if instead I just check that value column for anomalies with unsupervised learning. would that be an option and how would you go on doing so ? $\endgroup$ – be1995 Jan 23 at 17:36
  • $\begingroup$ Read my answer carefully. Anomaly is a label. Unsupervised = no label, Just internal patterns $\endgroup$ – Pieter21 Jan 23 at 18:38
  • $\begingroup$ you can detect anomalies using unsupervised learning. I have read quite some articles regarding this $\endgroup$ – be1995 Jan 23 at 18:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.