Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or outliers inside this csv file.
Being relatively new to this and not knowing much I would appreciate if I get some help or a small guidance and how I should tackle this problem. What methods or what is the best approach to find outliers in a Dataframe.
My language of choice is Python
You should plot the data with matplotlib and seaborn to get a visual view of the data. That is where I would start. Especially with large datasets, this is a quick way to visually see any outliers.
Detecting outliers is, unfortunately, more of an art than science. The famous statistician John Tukey proposed as IQR 1.5 as a “outlier”. Hence, the upper fence is 75% + (IQR 1.5).
Here's the code in Python for the feature "Balance":
Outliers are nothing but extreme values in the dataset. It can be either too high or too low. Easiest way of detecting outlier is by plotting box plot.
Please refer following horizontal box plot:
Example
You can use plotly express to plot interactive box chart.
Something that has helped me multiple times ... especially in Supervised problems like regression is called the Cook's Distance.
The exact problem cook's distance solves is that instead of solving the outlier problem at a single variable level, cook's distance measures the total distorting effect of an observation on the regression line. (Thus necessitating the existence of a regression line, hence a supervised model). The higher the value, the more effect deleting that observation has on the overall regression line. While there are thumb rules about what kind of cutoff values should be used, I tend to select observations by plotting the cooks distance of observations and deleting observations which are highly separated from the cluster of the other points. It usually leads me to have higher accuracy.
https://en.wikipedia.org/wiki/Cook%27s_distance (For understanding Cook's distance)
https://stats.stackexchange.com/questions/164099/removing-outliers-based-on-cooks-distance-in-r-language (For implementation of above method in R)
anomaly detection. Andrew Ng course has several videos on the subject. $\endgroup$ – Xaqron Jan 13 '20 at 11:22