I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each column are $0$. The data represents certain attributes about a customer, including the proportion of a certain customer's purchases in a particular category (so that the sum of each of these proportions is $1$), in addition to other data like the number of visits, the time between visits, etc. A set of sample rows could look like:

columns = [other_stuff,'C1','C2','C3','C4','C5','C6']
row = [1945,0.45, 0, 0, 0, 0.3, 0.25]
another_row = [438,0, 0.24, 0, 0.01, 0.5, 0.25]

A sample histogram of one of the "proportional" variables looks like:

Distribution of one of the weird variables

As of now, I am struggling to find ways to scale the data to make it usable for clustering with other data with much different units/orders of magnitude. Should I:

  1. Scale all the other variables to a 0-1 range using a Min-Max method and keep the "proportional" variables the same

  2. Scale all variables using the variables' means and standard deviations, even though the "proportional variables" clearly do not follow a normal distribution (also cannot apply a log transformation since most of the values are indeed 0)

  3. Keep everything as is, but perform dimensionality reduction on all variables and perform clustering based off the principal components (if Principal Component Analysis is used)

  4. None of the above; use a completely different set of algorithms/methods

I am currently using option 3 using sckit-learn and Python 3.7. If there are packages in R that could also help, please throw them my way. Thank you.

  • $\begingroup$ I don't really think that the standardization technique using means and standard deviations is done to really make variables standard nornals but to simply rescale the variable to be unitless variables with means 0 and variances 1. There is no assumption of normality when using this technique even though it closely resembles that of a z-score. If you aren't doing statisical inference, however, then this really isn't an issue. You aren't actually comparing the transformed columns to a standard normal 0,1. $\endgroup$
    – aranglol
    Jun 12 '19 at 18:44
  • $\begingroup$ @aranglol thank you for the quick reply. Your point is interesting because I indeed thought that the sole purpose of normalization was to bring all of the data to a standard normal, but perhaps that speaks to my level of knowledge regarding standardization. Thank you for the response $\endgroup$ Jun 12 '19 at 20:55

You don't specify, but assuming you are doing K-means clustering, the way the algorithm calculates similarity, or more technically distance, using Euclidean Distance, the distances on the dimension with the greatest magnitude will overwhelm distances of those where the values are orders of magnitude smaller.

To overcome this I would normalize the data. I am not sure the method you use is such a big deal. Use min-max if you are comfortable with it. Be careful though because it is possible, if I remember correctly, in sk-learn to scale the whole data set in one go which doesn't fix the magnitude differences. Scale each column individually or even possibly just the one that is vastly different if all the others are nominally between 0 and 1.

  • $\begingroup$ Yes, thank you for specifying this; I did not specifically mention k-means but I am using it. I like your approach of scaling each column individually rather than the whole-go, which is what I was doing before. Do you think that normalizing is still a good option even if the distribution is very skewed? Thank you! $\endgroup$ Jun 12 '19 at 20:52
  • $\begingroup$ Personally, I would not normalize the data that is between 0 and 1. Don’t get hung up on the fact that is does not fit the normal distribution. It is not an assumption required for the validity of k-means. The data you displayed does look pretty close to a Pareto distribution though. Perhaps reading a little bit about the Pareto distribution would be really informative and give you insight into how the data may impact your model/application. $\endgroup$
    – Skiddles
    Jun 12 '19 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.