0
$\begingroup$

I'm making some RFM Analyses (Customer Segmentation) and, in order to feed the RFM data to K-Means, I need to unskew the data, as K-Means works best when dealing with symmetrical distributions.

One of the best practices I've found so far is to use log transforms in order to unskew the data. However, after a bit of experimenting (and, hopefully, correct intuition), I've also found that PCA is working really well at unskewing the data, without the need of performing a log transform. Is this supposed to work or am I in danger of making incorrect assumptions?

On a (cough) safer bet, I was planning on using PCA after a log transform and the scaling of the data, since it would help to orthogonalize it and, I suppose, improve K-Means's performance. Is this better or worse than simply applying PCA without the log transform?

$\endgroup$

1 Answer 1

1
$\begingroup$

On a (cough) safer bet, I was planning on using PCA after a log transform and the scaling of the data, since it would help to orthogonalize it and, I suppose, improve K-Means's performance. Is this better or worse than simply applying PCA without the log transform?

This is your solution, this is the normal, most used and best practice on what to do with skewed data

$\endgroup$
5
  • $\begingroup$ It does indeed make more sense than the first option, but could you point out a reference? $\endgroup$
    – psygo
    Commented Apr 30, 2019 at 10:50
  • $\begingroup$ ncbi.nlm.nih.gov/pmc/articles/PMC4408558/#!po=1.36364 $\endgroup$ Commented Apr 30, 2019 at 11:11
  • $\begingroup$ Out of curiosity, how did you find that reference? Google? Any others would be more than welcome too. And, for completeness, I would recommend you add them to your answer . $\endgroup$
    – psygo
    Commented Apr 30, 2019 at 13:55
  • 1
    $\begingroup$ On a sidenote, there is also a PowerTransformer function in the preprocessing module of sklearn that helps with normalizing distributions. $\endgroup$
    – psygo
    Commented Apr 30, 2019 at 14:01
  • 1
    $\begingroup$ Yes! The Box-Cox set of transformations is a generalization of the log transformation with $\lambda = 0$ $\endgroup$ Commented Apr 30, 2019 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.