# Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the estimator in order to find outliers. However, I would like to get the probabilities that the point is an outlier. Can this be done in sklearn? Is there an R package to do it?

I don't even know if there is a theoretical foundation of the outlier detection methods used by sklearn that allows you to give probabilities. If not, what is the criterium that tells you what is an outlier and what is not? Does it consider probabilities or some kind of non-probabilistic scores?

Any piece of information will be appreciated.

Edit

I would like the outlier method to consider the multivariate distribution of the data. I think that univariate detection methods are rather poor.

• general anything beyond +-3std is considered as an outlier, or considering the IQR, Q1-1.5*IQR, Q3+1.5*IQR, that's what a boxplot does actually – Aditya May 31 '18 at 9:04
• And can you tune the 3 std deviations? What if I want to say that 2 standard deviations work for me? Can I do that in sklearn? – David Masip May 31 '18 at 9:45
• To be honest, 3rd STD will nearly have almost 99% of your data covers, so the remains are generally the outliers, but there's a problem here also, as whether an outlier is an outlier or an influential.. – Aditya Jun 1 '18 at 11:23

A simple trick to do outlier detection is to use the output probability of your model. If you are using a neural network for instance, you can use a softmax output which will give you a probability for each labels: $$p(y=y_i) = \frac{e^{W_i^Tx+b_i}}{\sum_j e^{W_j^Tx+b_j}}$$
If your model is accurate, for most points in your dataset, the probability should be peaked on the true label. However, if you have an outlier, then the model should get confused and then return probabilities that are more spread out over the labels. You could measure this spread by measuring the entropy of the output softmax probability distribution p(y): $$H[p] = - \sum_y p(y)\log p(y),$$ where $y$ takes one the different output categories (labels). Entropy is a measure of uncertainty, so if you have $H(p)=0$, the model is confident of the output, whereas if $H[p] = \log(N_c)$ ($N_c$ the number of labels), the model has no idea what to predict. Thus, using a threshold on the entropy (you might have to tune that threshold depending on your specific problem), you categorize a point as being an outlier or not. I think this may be a good starting point for high-dimensional data. For low-dimensional data, you can always do some density estimation and use the density as a threshold.