# How to scale or standardize data that is mostly 0 (ranges from 0-1)?

I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $$0-1$$, but most of the values in each column are $$0$$. The data represents certain attributes about a customer, including the proportion of a certain customer's purchases in a particular category (so that the sum of each of these proportions is $$1$$), in addition to other data like the number of visits, the time between visits, etc. A set of sample rows could look like:

columns = [other_stuff,'C1','C2','C3','C4','C5','C6']
row = [1945,0.45, 0, 0, 0, 0.3, 0.25]
another_row = [438,0, 0.24, 0, 0.01, 0.5, 0.25]



A sample histogram of one of the "proportional" variables looks like:

As of now, I am struggling to find ways to scale the data to make it usable for clustering with other data with much different units/orders of magnitude. Should I:

1. Scale all the other variables to a 0-1 range using a Min-Max method and keep the "proportional" variables the same

2. Scale all variables using the variables' means and standard deviations, even though the "proportional variables" clearly do not follow a normal distribution (also cannot apply a log transformation since most of the values are indeed 0)

3. Keep everything as is, but perform dimensionality reduction on all variables and perform clustering based off the principal components (if Principal Component Analysis is used)

4. None of the above; use a completely different set of algorithms/methods

I am currently using option 3 using sckit-learn and Python 3.7. If there are packages in R that could also help, please throw them my way. Thank you.

• I don't really think that the standardization technique using means and standard deviations is done to really make variables standard nornals but to simply rescale the variable to be unitless variables with means 0 and variances 1. There is no assumption of normality when using this technique even though it closely resembles that of a z-score. If you aren't doing statisical inference, however, then this really isn't an issue. You aren't actually comparing the transformed columns to a standard normal 0,1. – aranglol Jun 12 '19 at 18:44
• @aranglol thank you for the quick reply. Your point is interesting because I indeed thought that the sole purpose of normalization was to bring all of the data to a standard normal, but perhaps that speaks to my level of knowledge regarding standardization. Thank you for the response – CannedScientist Jun 12 '19 at 20:55