My data (continuous) is highly skewed and it doesn't follow the normal distribution. Using sns.distplot I found out that exponweib fits the data better.
How to deal with this?
My end goal is to use the data for machine learning model (SVM).
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
0
You’re fine to proceed without worrying about lacking normal data. Go run your SVM.
Even in linear regression, the frequent assumption about normality has to do with the error term. Further, that assumption is not part of prediction. The Gauss-Markov theorem does not assume a normal error term, so the $\hat{\beta}=(X^TX)^{-1}X^Ty$ parameter estimate is the best linear unbiased estimator whether the error term is normal or not.
When we do make an assumption about a normal error term, that is to help us with parameter inference, not prediction.
That’s on the side of the response variable, though. For the predictor variables, we absolutely do not make any assumptions about normality, not even for parameter inference.
So please feel free to run your SVM without worrying about your data lacking a normal distribution.
$\begingroup$
$\endgroup$
3
To address skewed data you can do data transformation like logarithmic transformation, squared transformation etc.
Alternately you could try non-parametric machine learning algorithms that does not have any assumption of normality of x variables.
Hope this will help....
-
$\begingroup$ Thanks for the help. Using non-parametric algorithms do sounds better. But still in future in faced with this I am not sure how to deal with it. I tried doing data transformation but since the range from .0 to 1 I was facing an error. $\endgroup$– MaximaJan 8, 2020 at 12:23
-
$\begingroup$ Hi @Swarley, i am not sure why and what type of error you are getting in transformation. Ideally it must work and have seen logarithmic transformation has been very useful in such scenarios. I recommend to resolve the error and try to fit algorithm again. $\endgroup$– SKBJan 8, 2020 at 13:15
-
$\begingroup$ My data has 0's which cant be avoided thats why using log the result is NaN $\endgroup$– MaximaJan 8, 2020 at 13:17
$\begingroup$
$\endgroup$
2
There are several techniques to treat with this data such as:
- Winsorization
- Clipping
- Removing
answered Jan 8, 2020 at 9:34
-
$\begingroup$ Thank you, The outliers in my data are valid. So I cant go ahead with Winsorization. I am not sure of Clipping and what does removing means here? $\endgroup$– MaximaJan 8, 2020 at 12:27
-
$\begingroup$ Yes, winsorization is a good technique for linear models. See if it improves the score. $\endgroup$ Jan 8, 2020 at 12:49
$\begingroup$
$\endgroup$
This seems pretty relevant for your specific scenario.
https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html
Consider pre-processing options as well. 1) rescale your data 2) standardize your data 3) normalize your data
See the link below for some (really simple) ways to do these things.
https://machinelearningmastery.com/prepare-data-machine-learning-python-scikit-learn/
That should do it!!
