Do you apply outlier detection of numerical data in practical applications?

Question

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data.

Do you apply outlier detection in your usual routine? By this I mean: among $N$ training data decide $K<<N$ data points to be outliers and discard them.

What are your experiences doing this? Which methods do you apply for numerical data.

Answer 1

You'd have to think about where the outliers come from. Technical malfunction? Then they are probably safe to disregard. But when you write that your main task is

to draw conclusions from the training data that can be generalized to future unseen data

you have to keep in mind that your future data will contain outliers as well and deal with them appropriately. Models like trees are usually somewhat robust against those while regression can go haywire. There are several well working methods like disregarding everything that is more than 4 standard deviations (or median deviations from the median) out or just disregarding the n-th top and bottom percentiles.

But then there is the case where outliers are not just flukes and can actually tell you something about the problem domain you are working on. If you work in fraud detection, the outliers are the points that interest you. If you work on optimizing network performance, the packets that don't arrive can be the interesting ones.

So my advice is to not blindly throw away data out of principle but always think about what the data represent and why outliers may be present.

Answer 2

As far as practical applications are concerned :

Here are a few:

1) Aircraft Engine Anomaly Detection: Input Features can be heat generated by engine, vibrational intensity, fuel consumed etc etc. Here outliers can be sent for testing again and further decisions can be made.

2) Fraud Detection : Features can be features of users activities on a website. We can model probabilities from the data. Identify unusual behavior by checking probability less than certain fixed threshold.

3) Monitoring Computers in a data center : features can be memory use, no of disk accesses, CPU load, network traffic etc. Abnormal behavior here can help predict future breakdowns.

Anomaly Detection is done assuming our data has a probability distribution(gaussian). We can plot data to see if thats the case, if not we can make it gaussian using log transforms. Gaussian distribution specifies the regions and probabilities of our data lying in those regions.

For example : replace original feature x -> Log(x) or feature x -> (x)^4/3 etc..

Also regarding the threshold value which decides outliers you can play with it and see that with Higher threshold you will be rejecting more entries and this might be required where doctors are trying to isolate cancer patients amongst many normal ones without taking any risk/chances. Again outliers here doesn't mean cancer patient but definitely worth a medical test. And you can set it to lower value if you are getting too many normal data flagged as outliers.

We have skewed data sets since we have more examples of one kind than the other. For example when we get air craft engine data we might just have data for few bad ones and mostly for good ones.Use of cross validation data is suggested.F1-Score is a pretty good metric to evaluate the performance of the algorithm.

Answer 3

If you are dealing with continuous data that generally fits a normal distribution, you can apply the Z-score as a method of determining how many standard deviations the observed value is from the mean. Depending on your objective, you could throw out large z-scores. https://en.wikipedia.org/wiki/Standard_score

Boxplots are good for visualizing and identifying outliers, although just a boxplot can sometimes hide information about the data.

Here is a link to a similar conversation: Which algorithms or methods can be used to detect an outlier from this data set?