3
$\begingroup$

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or outliers inside this csv file.

Being relatively new to this and not knowing much I would appreciate if I get some help or a small guidance and how I should tackle this problem. What methods or what is the best approach to find outliers in a Dataframe.

My language of choice is Python

$\endgroup$
  • $\begingroup$ You are looking for anomaly detection. Andrew Ng course has several videos on the subject. $\endgroup$ – Xaqron Jan 13 '20 at 11:22
2
$\begingroup$

You should plot the data with matplotlib and seaborn to get a visual view of the data. That is where I would start. Especially with large datasets, this is a quick way to visually see any outliers.

$\endgroup$
  • $\begingroup$ thank you sounds good. The final goal is to make a data checker if I can call it that way. It checks certain columns in from that dataframe in csv file for outliers and if there is any it tell you which ones. Any advice ? I am also not sure what machine learning algorithms would be good for this . $\endgroup$ – be1995 Jan 12 '20 at 19:40
1
$\begingroup$

Detecting outliers is, unfortunately, more of an art than science. The famous statistician John Tukey proposed as IQR 1.5 as a “outlier”. Hence, the upper fence is 75% + (IQR 1.5).

Here's the code in Python for the feature "Balance":

enter image description here

$\endgroup$
  • $\begingroup$ I asked for what would be the first steps to take if I want to end up with an anomaly detection at the end. Which machine learning allgorithms are where should I start with. I know this is a broad topic that's why I ended up here for some advice $\endgroup$ – be1995 Jan 13 '20 at 8:06
1
$\begingroup$

Outliers are nothing but extreme values in the dataset. It can be either too high or too low. Easiest way of detecting outlier is by plotting box plot.

Please refer following horizontal box plot:

enter image description here

enter image description here

Example

You can use plotly express to plot interactive box chart.

enter image description here

$\endgroup$
0
$\begingroup$

Something that has helped me multiple times ... especially in Supervised problems like regression is called the Cook's Distance.

The exact problem cook's distance solves is that instead of solving the outlier problem at a single variable level, cook's distance measures the total distorting effect of an observation on the regression line. (Thus necessitating the existence of a regression line, hence a supervised model). The higher the value, the more effect deleting that observation has on the overall regression line. While there are thumb rules about what kind of cutoff values should be used, I tend to select observations by plotting the cooks distance of observations and deleting observations which are highly separated from the cluster of the other points. It usually leads me to have higher accuracy.

https://en.wikipedia.org/wiki/Cook%27s_distance (For understanding Cook's distance)

https://stats.stackexchange.com/questions/164099/removing-outliers-based-on-cooks-distance-in-r-language (For implementation of above method in R)

$\endgroup$
  • $\begingroup$ from statsmodels.formula.api import ols m = ols(<formula>,<data>).fit() infl = m.get_influence() sm_fr = infl.summary_frame() Lines above represent a way to get cook's distance in python. $\endgroup$ – Vineet Agarwal Jan 13 '20 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.