0
$\begingroup$

Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my data, which are not possible to be excluded manually.

Now my question is, How is it possible to convert this dataset into a gaussian (zero skewness) so that a learning procedure could be run on it without being biased to outliers? How should one behave this kind of datasets?

$\endgroup$

1 Answer 1

0
$\begingroup$

I recommend order normalization from R package bestNormalize, good info at: https://cran.r-project.org/web/packages/bestNormalize/vignettes/bestNormalize.html

Below a short comparison of a few methods:

Best Normalizing transformation with 6283 Observations
Estimated Normality Statistics (Pearson P / df, lower => more normal):
 - No transform: 21.8352 
 - Box-Cox: 4.0105 
 - Log_b(x+a): 4.5358 
 - sqrt(x+a): 7.1273 
 - arcsinh(x): 4.5268 
 - Yeo-Johnson: 4.0185 
 - orderNorm: 0.9744 

It is not written anywhere, but logic says, that you should rbind train and test datasets before this normalization if you want to use all of its advantages.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.