# Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or outliers inside this csv file.

Being relatively new to this and not knowing much I would appreciate if I get some help or a small guidance and how I should tackle this problem. What methods or what is the best approach to find outliers in a Dataframe.

My language of choice is Python

• You are looking for anomaly detection. Andrew Ng course has several videos on the subject. Jan 13, 2020 at 11:22

You should plot the data with matplotlib and seaborn to get a visual view of the data. That is where I would start. Especially with large datasets, this is a quick way to visually see any outliers.

• thank you sounds good. The final goal is to make a data checker if I can call it that way. It checks certain columns in from that dataframe in csv file for outliers and if there is any it tell you which ones. Any advice ? I am also not sure what machine learning algorithms would be good for this . Jan 12, 2020 at 19:40

Detecting outliers is, unfortunately, more of an art than science. The famous statistician John Tukey proposed as IQR 1.5 as a “outlier”. Hence, the upper fence is 75% + (IQR 1.5).

Here's the code in Python for the feature "Balance":

• I asked for what would be the first steps to take if I want to end up with an anomaly detection at the end. Which machine learning allgorithms are where should I start with. I know this is a broad topic that's why I ended up here for some advice Jan 13, 2020 at 8:06

Outliers are nothing but extreme values in the dataset. It can be either too high or too low. Easiest way of detecting outlier is by plotting box plot.

Please refer following horizontal box plot:

Example

You can use plotly express to plot interactive box chart.

Something that has helped me multiple times ... especially in Supervised problems like regression is called the Cook's Distance.

The exact problem cook's distance solves is that instead of solving the outlier problem at a single variable level, cook's distance measures the total distorting effect of an observation on the regression line. (Thus necessitating the existence of a regression line, hence a supervised model). The higher the value, the more effect deleting that observation has on the overall regression line. While there are thumb rules about what kind of cutoff values should be used, I tend to select observations by plotting the cooks distance of observations and deleting observations which are highly separated from the cluster of the other points. It usually leads me to have higher accuracy.

https://en.wikipedia.org/wiki/Cook%27s_distance (For understanding Cook's distance)

https://stats.stackexchange.com/questions/164099/removing-outliers-based-on-cooks-distance-in-r-language (For implementation of above method in R)

• from statsmodels.formula.api import ols m = ols(<formula>,<data>).fit() infl = m.get_influence() sm_fr = infl.summary_frame() Lines above represent a way to get cook's distance in python. Jan 13, 2020 at 10:25