2
$\begingroup$

I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit.

I'm trying to find possible outliers in all these dimensions. I used Z-score to detect outliers in single dimension to find which high cost is causing outliers.

As a next step I tried to find outliers with high cost and high profit and low discount.

I came up with a formula of :

Zscore(cost) + Zscore(profit) - Zscore(discount)

negative sign because I want to find outliers with low discount.

Is this approach meaningful to do? or is there any further proven way to achieve this?

$\endgroup$
1
$\begingroup$

Have you considered using the Mahalanobis Distance? It's can be thought of as the magnitude of a multi-dimensional Z-score.

The way I prefer to view the the Mahalanobis Distance is as the square root of the exponent of the Multivariate Normal Distribution. This is similar to the Z-score, which is the exponent of the univarate normal distribution before the square is applied. A big difference between these two is the Z-score is signed, while the Mahalanobis Distance is unsigned, which doesn't matter for finding outliers anyway.

I understand you don't have normally distributed data, but sometimes pretending your data is normally distributed can have good results, so using the Mahalanobis Distance can be worth investigating.

$\endgroup$
  • $\begingroup$ sounds interesting could you please elaborate on this $\endgroup$ – tourist Jun 30 '16 at 7:08
  • $\begingroup$ I added a bit more, explanation, hope it helps! $\endgroup$ – David Garwin Jun 30 '16 at 15:23
0
$\begingroup$

Because your data is normal distributed ( gaussian) you could easy try to implement in your desired language this alghoritm from coursera mooc : https://class.coursera.org/ml-005/lecture/91

$\endgroup$
  • $\begingroup$ but the question said the data is not normal? $\endgroup$ – Marcus D Apr 30 '16 at 9:11
  • 1
    $\begingroup$ Yeah I notticed that now , sorry , I have not read the didn't $\endgroup$ – Daia Alexandru Apr 30 '16 at 9:39
0
$\begingroup$

Using Z-score can be ok if you're sure about what you're looking for. It can also be just a way to transform your data before using some ML on it. Be aware that Z-score applies to normally distributed data (which you say is not the case). Be also aware that looking for outliers in 3 dimensions is not as simple as looking 3 times for outliers in 1 dimension. You should plot your data in 3D, and try to find where might be the outliers.

Otherwise, one-class SVMs are pretty good at anomaly/outliers detection. Take a look at the introduction here. Also, any clustering algorithm might be helpful to get a better insight. DBSCAN, for example, helps you to find clusters based on the density of the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.