# Detecting outlier with combining two vectors

I want to combine the following vectors in a way that just the red point (number 7) becomes inconsistent with other points( become an outlier and become distant from other points) and other points become consistent with each other. Please note that I have tested mahalanobis distance and Kullback-Leibler divergence between two vectors but they were not so good and detects.

a=[1.3269 1.3354 1.3318 1.3282 1.34666 1.3460 1.36084 1.3526 1.3539 1.3510 1.3480 1.3479 1.34893]

b=[0.0352,0.0992,0.1570,0.1431,0.1634,0.1629,0.1046,0.1655,0.1635,0.1642,0.1658,0.1666,0.15735]

From the second graph it seems pretty easy to identify the outlier. You could probably just fit a simple polynomial (or some other function) and then flag all points that have a distance greater than 2 standard deviations (or whatever seems appropriate) from the fitted curve.

import numpy as np
import matplotlib.pyplot as plt

b=np.array([0.0352,0.0992,0.1570,0.1431,0.1634,0.1629,0.1046
,0.1655,0.1635,0.1642,0.1658,0.1666,0.15735])
x = np.arange(13)
p = np.poly1d(np.polyfit(x,b,4))
y = p(x)

plt.plot(x,b,'ro')
plt.plot(x,y,'b-')
plt.show()

• I don't think the horizontal axis has any significance, which is what you are using to create the deviation for point 6. In 1-D is less of an outlier than points 0 and 1. Apr 26 '17 at 4:53
• Thanks, oW_. It works and I really liked your approach. But please note that the vector is one dimension. The values of the vectors are readings of different sensors in a plant (they measure the same thing but in different sections). each sensor has a number and the values in the vectors are based on that order. Now, please let me know that if one fit a curve on a multivariate vector, will be there any debate on the usage? Apr 26 '17 at 6:13
• Based on this comment, I would suggest some additional feature engineering. Is there an assumption of isotropy in space or could you add sensor location as an additional feature? Sensors have time histories, so can you turn the time series into rectangular data. Derivatives and integrals in time? Whats going on in the plant? Can you add day/night features, number of employees present, number of cogs being produced when each datapoint is produced? Data is king, so feature engineering can add loads of signal for an algorithm to digest. Apr 26 '17 at 11:48
• Thanks. But this vector is the output of doing some feature engineering on the primary vectors and it is not possible to do more. Now I need to detect outlier between this vector to say that the sensor i-th (x-axis) has some invalid values. In the second step, I need to further investigate and recognize the outlier values between the values of that sensor (a sensor). Now you think if we fit a curve on this problem it will sound unreasonable and debatable? (it solve our problem but being justifiable and logical is so imotnant for me) Apr 26 '17 at 12:36
• ok, sorry that wasn't clear to me. then this approach doesn't make much sense...
– oW_
Apr 26 '17 at 16:51

I don't really agree with the idea of "wanting" a point to be an outlier and then massaging the algorithm to make it so. You have 2 dimensions and its either an outlier or its not.

If one standardizes the data and then deciphers the Mahalanobis distances, point 6 is only one of two points that sit outside of a certain threshold (point 0 being the other point). Beyond that theres not much you can do beyond some sort of nonlinear transformation which you know to be true due to some deterministic knowledge you have about a particular phenomenon.

Anyways... here's the two outlier version in case you weren't standardizing your data first:

from pandas import DataFrame, read_csv
df=(dfR-dfR.mean())/dfR.std()

import numpy as np
import matplotlib.pyplot as plt
from sklearn.covariance import EmpiricalCovariance, MinCovDet
emp_cov = EmpiricalCovariance().fit(df)

fig = plt.figure()
plt.subplots_adjust(hspace=-.1, wspace=.4, top=.95, bottom=.05)

# Show data set
subfig1 = plt.subplot(3, 1, 1)
my_plot = subfig1.scatter(df.A,df.B)
subfig1.set_xlim(subfig1.get_xlim()[0], 11.)
subfig1.set_title("Mahalanobis distances")

# Show contours of the distance functions
xx, yy = np.meshgrid(np.linspace(plt.xlim()[0], plt.xlim()[1], 100),
np.linspace(plt.ylim()[0], plt.ylim()[1], 100))
zz = np.c_[xx.ravel(), yy.ravel()]

mahal_emp_cov = emp_cov.mahalanobis(zz)
mahal_emp_cov = mahal_emp_cov.reshape(xx.shape)
emp_cov_contour = subfig1.contour(xx, yy, np.sqrt(mahal_emp_cov),
cmap=plt.cm.PuBu_r,
linestyles='dashed')

subfig1.legend([emp_cov_contour.collections[1],
my_plot],
['MLE dist'],
plt.xticks(())
plt.yticks(())


Hope this helps!

• Thanks. I really appreciate your reply. Do you standardize each vector (I mean using the means and standard deviations along the row)? Because when I standardize each vector and then calculate MLE between vectors and visualize the new vector, the red point is consistent with other ones. Apr 26 '17 at 6:29
• Yes, standardize each vector. This gets to the distinction between outlier vs. novelty detection vs. classification. You are correct that your red point is within two standard deviations along each dimension. However one could certainly treat this as a supervised learning problem and pre-label point 7 as distinct. In this case, you don't have much data to go by, so this becomes a tough problem. The set is small enough that robust statistics likely make more sense than mean and stdev. Again, not asserting that I'm providing much use, but just that there are plausible decision surfaces present. Apr 26 '17 at 11:39
• Thanks. but I need to use an unsupervised method since the data distribution changes. The red point is just different when we see the trend od data like the fitted curve. I'm wondering that is there any way to take advantage of this point and solve the problem in an unsupervised manner. Apr 26 '17 at 12:49
• @Arkan "trend"? Is it a time series or not? what makes you determine that 6 is an outlier but not 0? because if it is not a time series then I don't think it can be justified with the data you provided...
– oW_
Apr 26 '17 at 17:06
• No, by trend, I mean the fitted curve or line. I mean to not use curve fitting as it is a one dimension but can use or reach a similar idea to solve this problem Apr 26 '17 at 19:04

## Autoencoder Solution

You could try an autoencoder. The autoencoder would take an input vector and it would try to recreate it as an output. So you take your input, and measure the distance between the input and the predicted output variable, using your metric of choice (euclidean should work but try various). Larger distances can be thought of as more abnormal. So you can stack rank your observations from weirdest to most normal.

Make sure that you are only training the autoencoder on normal data though. This would of course assume that you have more than the 13 samples that you are looking at. If not this probably isn't going to work very well, just because of the small sample.

## KDE Solution

The idea is to use Kernel Density Estimation to generate a non-parametric joint density of your data set. You then find what the probability of finding a value that extreme would be. Here is some code using python's sklearn package:

from sklearn.neighbors.kde import KernelDensity
import numpy as np
X=np.matrix([[1.3269, 1.3354, 1.3318, 1.3282, 1.34666, 1.3460, 1.36084, 1.3526, 1.3539, 1.3510, 1.3480, 1.3479, 1.34893],[0.0352, 0.0992, 0.1570, 0.1431, 0.1634, 0.1629, 0.1046, 0.1655, 0.1635, 0.1642, 0.1658, 0.1666, 0.15735]])
kde = KernelDensity(kernel='gaussian', bandwidth=.45).fit(X.T)
score=kde.score_samples(X.T)
prob=np.exp(score)
print(prob/prob[6])


This code shows that the observations in the lowest probability density areas are observations 1,2 and 7. Of course this would work better with a larger sample, and you need to fuss with the bandwidth to calibrate it, but that this should do it.

• Thanks. But I need an unsupervised method. I think the autoencoders needs the class label. Apr 26 '17 at 6:06
• You seem to already know that it is different, this is where labelled data comes from. It is tedious and time consuming but you can manually verify that the data is good before feeding it in to the autoencoder. Having read through some of the suggestions, and your response, I think you could try fitting a kernel density estimator, and just pulling the probability of a point that extreme. It is non-parametric, and unsupervised, but it would require more data than it sounds like you have to work well.
– Ryan
Apr 26 '17 at 15:50
• Thanks. Can you explain a bit more about KDE. Should I use multivariate KDE considering my vector? Can I run it one a vector or as you said we should have a matrix. As I know if we use KDE it will return the density at different points (for example 100 points). So, how can we detect the extreme point in it? Apr 26 '17 at 16:16
• I added a KDE solution to my answer. It would still work better with a larger sample.
– Ryan
Apr 26 '17 at 21:14
• Thank you so much for the added answer. Should we run univariate kde for this problem? Because the vector values all are the same parameter but for different sections captured by different sensors. This function do univariate? I guess there are some automatic rules for obtaining bandwidth values such thumb rule., etc. I also tested your code it is similar to the second graph which points 1, 2, 7 have a smaller value. Apr 27 '17 at 5:57