2
$\begingroup$

My data (continuous) is highly skewed and it doesn't follow the normal distribution. Using sns.distplot I found out that exponweib fits the data better.

How to deal with this?

My end goal is to use the data for machine learning model (SVM).

$\endgroup$
2
$\begingroup$

To address skewed data you can do data transformation like logarithmic transformation, squared transformation etc.

Alternately you could try non-parametric machine learning algorithms that does not have any assumption of normality of x variables.

Hope this will help....

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the help. Using non-parametric algorithms do sounds better. But still in future in faced with this I am not sure how to deal with it. I tried doing data transformation but since the range from .0 to 1 I was facing an error. $\endgroup$ – Tangent Jan 8 at 12:23
  • $\begingroup$ Hi @Swarley, i am not sure why and what type of error you are getting in transformation. Ideally it must work and have seen logarithmic transformation has been very useful in such scenarios. I recommend to resolve the error and try to fit algorithm again. $\endgroup$ – SKB Jan 8 at 13:15
  • $\begingroup$ My data has 0's which cant be avoided thats why using log the result is NaN $\endgroup$ – Tangent Jan 8 at 13:17
1
$\begingroup$

There are several techniques to treat with this data such as:

  1. Winsorization
  2. Clipping
  3. Removing
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, The outliers in my data are valid. So I cant go ahead with Winsorization. I am not sure of Clipping and what does removing means here? $\endgroup$ – Tangent Jan 8 at 12:27
  • $\begingroup$ Yes, winsorization is a good technique for linear models. See if it improves the score. $\endgroup$ – Carlos Mougan Jan 8 at 12:49
0
$\begingroup$

This seems pretty relevant for your specific scenario.

https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html

Consider pre-processing options as well. 1) rescale your data 2) standardize your data 3) normalize your data

See the link below for some (really simple) ways to do these things.

https://machinelearningmastery.com/prepare-data-machine-learning-python-scikit-learn/

That should do it!!

| improve this answer | |
$\endgroup$
0
$\begingroup$

You’re fine to proceed without worrying about lacking normal data. Go run your SVM.

Even in linear regression, the frequent assumption about normality has to do with the error term. Further, that assumption is not part of prediction. The Gauss-Markov theorem does not assume a normal error term, so the $\hat{\beta}=(X^TX)^{-1}X^Ty$ parameter estimate is the best linear unbiased estimator whether the error term is normal or not.

When we do make an assumption about a normal error term, that is to help us with parameter inference, not prediction.

That’s on the side of the response variable, though. For the predictor variables, we absolutely do not make any assumptions about normality, not even for parameter inference.

So please feel free to run your SVM without worrying about your data lacking a normal distribution.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Pardon me, I am beginner, but if faced with highly skewed data, I shouldn't worry much? If removing the skewness increases my model accuracy then shouldn't we go for it? $\endgroup$ – Tangent Mar 2 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.