2
$\begingroup$

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the estimator in order to find outliers. However, I would like to get the probabilities that the point is an outlier. Can this be done in sklearn? Is there an R package to do it?

I don't even know if there is a theoretical foundation of the outlier detection methods used by sklearn that allows you to give probabilities. If not, what is the criterium that tells you what is an outlier and what is not? Does it consider probabilities or some kind of non-probabilistic scores?

Any piece of information will be appreciated.

Edit

I would like the outlier method to consider the multivariate distribution of the data. I think that univariate detection methods are rather poor.

$\endgroup$
  • $\begingroup$ general anything beyond +-3std is considered as an outlier, or considering the IQR, Q1-1.5*IQR, Q3+1.5*IQR, that's what a boxplot does actually $\endgroup$ – Aditya May 31 '18 at 9:04
  • $\begingroup$ And can you tune the 3 std deviations? What if I want to say that 2 standard deviations work for me? Can I do that in sklearn? $\endgroup$ – David Masip May 31 '18 at 9:45
  • $\begingroup$ To be honest, 3rd STD will nearly have almost 99% of your data covers, so the remains are generally the outliers, but there's a problem here also, as whether an outlier is an outlier or an influential.. $\endgroup$ – Aditya Jun 1 '18 at 11:23
0
$\begingroup$

One of the easiest and quick way would be to plot a 'boxplot' where the "dots" above the upper whisker(T shaoed line) of the boxplot denotes an outlier(refer to the attached image). I hope this was what you are searching for. Do let me know if otherwise, may be I can help.enter image description here

$\endgroup$
  • $\begingroup$ I think that this method oversimplifies the problem, as it only considers the marginal distributions. $\endgroup$ – David Masip Jun 1 '18 at 6:02
  • $\begingroup$ You can say that, but as I mentioned its just one of the quick ways to visualize the outliers. For pointing out probabilties of outliers you can use the "quantile" equations stated by Aditya in his comment and devise a custom function to determine the probability $\endgroup$ – sapal6 Jun 1 '18 at 6:10
0
$\begingroup$

A simple trick to do outlier detection is to use the output probability of your model. If you are using a neural network for instance, you can use a softmax output which will give you a probability for each labels: \begin{equation} p(y=y_i) = \frac{e^{W_i^Tx+b_i}}{\sum_j e^{W_j^Tx+b_j}} \end{equation}

If your model is accurate, for most points in your dataset, the probability should be peaked on the true label. However, if you have an outlier, then the model should get confused and then return probabilities that are more spread out over the labels. You could measure this spread by measuring the entropy of the output softmax probability distribution p(y): \begin{equation} H[p] = - \sum_y p(y)\log p(y), \end{equation} where $y$ takes one the different output categories (labels). Entropy is a measure of uncertainty, so if you have $H(p)=0$, the model is confident of the output, whereas if $H[p] = \log(N_c)$ ($N_c$ the number of labels), the model has no idea what to predict. Thus, using a threshold on the entropy (you might have to tune that threshold depending on your specific problem), you categorize a point as being an outlier or not. I think this may be a good starting point for high-dimensional data. For low-dimensional data, you can always do some density estimation and use the density as a threshold.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.