# What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?

I am solving for a regression use case using tensorflow's DNNRegressor. For EDA purpose, I referred to this post and used pandas boxplot to plot my numerical predictors and target variable(here, pid demand) and scatter_matrix for plotting the distributions and here are the results : predictor_target_boxplot ; features_label_pdf_scatter_matrix .

I need help in interpreting these two plots, specifically on these fronts:

1. How come the boxplot shows so many points beyond whiskers (~10%), can there be so many outliers in a dataset?
2. How do I handle those outliers?
3. Based on the second plot (feature, label pdf), should I normalize my features to exhibit Gaussian distribution? If so, why?
• Dataset looks interesting, is it open sourced? Can you add a correlation plot also? Also try adding $lmplots$, I had come across a similar kind of dataset, In that case I didn't focus much on boxplots as the dataset was related to the price of Houses(Singapore Houses was the dataset), so it makes sense that prices now are way more than before and the box plots did show that – Aditya Apr 23 '18 at 11:49
• People now don't care about outliers anymore , thanks to the Xgboost, Also your distribution are skewed, use some log transformations – Aditya Apr 23 '18 at 12:00
• Thanks for all your inputs. Yes, XGBoost is pretty robust to outliers but I am using dnn regression as of now so was concerned as to how should I handle outliers. You have asked to add lmplots, may I know the significance? The data is not open-sourced, it's from a private project:) – chetna bansal Apr 23 '18 at 15:14
• to draw more conclusions, if you have a categorical variables.., then you should attempt.. – Aditya Apr 23 '18 at 16:26

This comments aren't all mine, I have asked on a slack forum,

boxplot is shouting at you: skewness, and also high dispersion. Can not ask much more to a boxplot than location, dispersion and skewness..

Also checkout this term heteroscedasticity(completely suits your case)

Try switching the transformation to a log plot or lower..

Also your eda can't be boxplots dependent as prices are involved here..one of the remedy would be doing box-cox transformation

https://www.differencebetween.com/difference-between-dispersion-and-vs-skewness/

http://www.statsmakemecry.com/smmctheblog/confusing-stats-terms-explained-heteroscedasticity-heteroske.html

To have a look at it, https://datascienceplus.com/how-to-detect-heteroscedasticity-and-rectify-it/