# How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?

I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(specifically STALE FILE HANDLEs for now in GPFS file system). I am categorizing the log files and clustering based on their instances per time interval. Since some messages are more predominant over the others in any given time frame than the others, i don’t want the clustering to bias towards the one with maximum variance.

• This is unclear as to what outcome you are interested in and what your data looks like. Example data and some data summaries would be useful. Without some knowledge of the data your question is basically unanswerable and we may as well suggest you add the phase of the moon to the square root of the Dow Jones index multiplied by your data. – Spacedman Jul 1 '16 at 7:23
• The main focus is to predict the occurrences of Stale File Handle.The log files consists of the error messages generated with time stamps, source node and few others. On first pass, i tried to use hierarchical clustering to see if all the 'abnormal' states(of time, grouped by occurrences per hour ) clustered together. But that becomes a problem because there are certain messages like 'connect to'/'connecting to' which are more predominant than the other and therefore hclustering without scaling results in bias towards the counts with maximum variance. – Kraamed Jul 5 '16 at 15:35

Its unclear what the OP is asking (so this response is somewhat general), but the table below illustrates common contexts and the transformations that are typical:

sales, revenue, income, price --> log(x)

distance --> 1/x, 1/x^2, log(x)

market share, preference share --> (e^x)/(1+e^x)

right-tailed dist --> sqrt(x), log(x) caution log(x<=0)

left-tailed dist --> x^2

You can also use John Tukey's three-point method as discussed in this post. When specific transformations don't work, use Box-Cox transformation. Use package car to lambda <- coef(powerTransform()) to compute lambda and then call bcPower() to transform. Consider Box-Cox transformations on all variables with skewed distributions before computing correlations or creating scatterplots.

• If questions are unclear you should add a comment and wait for the poster to clarify. – Spacedman Jul 1 '16 at 7:22
• I did use the Box-Cox. But isn't box cox used for normalizing the distribution of a uni-variate data. I am looking for a transformation that can be applied across the multivariate data, so as to remove bias. I did use the log(x) transformation. But i guess this question is more subjective, pertaining to the particular dataset i have. I am just having a hard time determining which transformation is the best. – Kraamed Jul 5 '16 at 15:40