25
$\begingroup$

I was reading about the solution to this OTTO Kaggle challenge and the first place solution seems to use several transforms for the input data X, for example Log(X+1), sqrt( X + 3/8), etc. Is there a general guideline on when to apply which kind transformations to various classifiers?

I do understand the concepts of mean-var and min-max normalization. However, for the above transformations, my guess is that Log and Sqrt are used to compress the dynamic range of the data. And the x-axis shift is just to recenter the data. However, the author chooses to use different methods of normalization for the same input X when feeding into different classifiers. Any ideas?

$\endgroup$
2
  • 1
    $\begingroup$ No idea what could possibly suggest that kind of formulas, but you might want to look at box-cox transformations, which suggest exponents for variables. $\endgroup$ Jul 29, 2017 at 6:28
  • $\begingroup$ Same here - no idea, have not seen this before. I guess they tried different transformations and picked the one that worked best. Since in the report they say that some other transformations would be also fine. $\endgroup$
    – Tobi
    Jul 31, 2017 at 13:21

2 Answers 2

22
$\begingroup$

We love the normal form

In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view!

Which Transformation?

The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions.

What makes physical (biological, economic, whatever) sense, for example in terms of limiting behaviour as values get very small or very large? This question often leads to the use of logarithms.

Can we keep dimensions and units simple and convenient? If possible, we prefer measurement scales that are easy to think about.

The cube root of a volume and the square root of an area both have the dimensions of length, so far from complicating matters, such transformations may simplify them. Reciprocals usually have simple units, as mentioned earlier. Often, however, somewhat complicated units are a sacrifice that has to be made.

When to Use What?

The most useful transformations in introductory data analysis are the reciprocal, logarithm, cube root, square root, and square. In what follows, even when it is not emphasised, it is supposed that transformations are used only over ranges on which they yield (finite) real numbers as results.

  • Reciprocal: The reciprocal, x to 1/x, with its sibling the negative reciprocal, x to -1/x, is a very strong transformation with a drastic effect on distribution shape. It can not be applied to zero values. Although it can be applied to negative values, it is not useful unless all values are positive. The reciprocal of a ratio may often be interpreted as easily as the ratio itself: Example:
    • population density (people per unit area) becomes area per person
    • persons per doctor becomes doctors per person
    • rates of erosion become time to erode a unit depth

(In practice, we might want to multiply or divide the results of taking the reciprocal by some constant, such as 1000 or 10000, to get numbers that are easy to manage, but that itself has no effect on skewness or linearity.)

The reciprocal reverses order among values of the same sign: largest becomes smallest, etc. The negative reciprocal preserves order among values of the same sign.


  • Logarithm: The logarithm, x log10 x, or x log e x or ln x, or x log 2 x, is a strong transformation with a major effect on distribution shape. It is commonly used for reducing right skewness and is often appropriate for measured variables. It can not be applied to zero or negative values. One unit on a logarithmic scale means a multiplication by the base of logarithms being used. Exponential growth or decline.

    • $y = a exp (bx)$

is made linear by - $ln y = ln a + bx$ so that the response variable y should be logged. (Here exp() means raising to the power e, approximately 2.71828, that is the base of natural logarithms). An aside on this exponential growth or decline equation: $x = 0$, and $y = a exp(0) = a$ so that a is the amount or count when x = 0. If a and b > 0, then y grows at a faster and faster rate (e.g. compound interest or unchecked population growth), whereas if a > 0 and b < 0, y declines at a slower and slower rate (e.g. radioactive decay).


  • Power functions :
  • $y = ax^b$ are made linear by $log y = log a + b log x$ so that both variables y and x should be logged. An aside on such power
    functions: put $x = 0$, and for $b > 0$,

  • $y = ax^b = 0$ so the power function for positive b goes through the origin, which often makes physical or biological or economic sense. Think: does zero for x imply zero for y? This
    kind of power function is a shape that fits many data sets
    rather well.

    • Consider ratios y = p / q where p and q are both positive in practice.
  • Examples are:

    • Males / Females
    • Dependants / Workers
    • Downstream length / Downvalley length
  • Then y is somewhere between 0 and infinity, or in the last case, between 1 and infinity. If p = q, then y = 1. Such definitions often lead to skewed data, because there is a clear lower limit and no clear upper limit. The logarithm, however, namely

  • log y = log p / q = log p - log q, is somewhere between -infinity and infinity and p = q means that log y = 0. Hence the logarithm of such a ratio is likely to be more symmetrically distributed.


  • Cube root: The cube root, x 1/3. This is a fairly strong transformation with a substantial effect on distribution shape: it is weaker than the logarithm. It is also used for reducing right skewness, and has the advantage that it can be applied to zero and negative values. Note that the cube root of a volume has the units of a length. It is commonly applied to rainfall data.

    • Applicability to negative values requires a special note. Consider
      (2)(2)(2) = 8 and (-2)(-2)(-2) = -8. These examples show that the
      cube root of a negative number has negative sign and the same
      absolute value as the cube root of the equivalent positive number. A similar property is possessed by any other root whose power is the
      reciprocal of an odd positive integer (powers 1/3, 1/5, 1/7, etc.)

    • This property is a little delicate. For example, change the power just a smidgen from 1/3, and we can no longer define the result as a product of precisely three terms. However, the property is there to be exploited if useful.


  • Square root:The square root, x to $x^(1/2)$ = sqrt(x), is a transformation with a moderate effect on distribution shape: it is weaker than the logarithm and the cube root. It is also used for reducing right skewness, and also has the advantage that it can be applied to zero values. Note that the square root of an area has the units of a length. It is commonly applied to counted data, especially if the values are mostly rather small.

  • Square: The square, x to $x^2$, has a moderate effect on distribution shape and it could be used to reduce left skewness. In
    practice, the main reason for using it is to fit a response by a
    quadratic function $y = a + b x + cx^2$. Quadratics have a turning
    point, either a maximum or a minimum, although the turning point in a function fitted to data might be far beyond the limits of the
    observations. The distance of a body from an origin is a quadratic if that body is moving under constant acceleration, which gives a very
    clear physical justification for using a quadratic. Otherwise
    quadratics are typically used solely because they can mimic a
    relationship within the data region. Outside that region they may
    behave very poorly, because they take on arbitrarily large values for extreme values of x, and unless the intercept a is constrained to be 0, they may behave unrealistically close to the origin.
    • Squaring usually makes sense only if the variable concerned is zero or positive, given that $(-x)^2$ and $x^2$ are identical.

$\endgroup$
5
  • $\begingroup$ thanks for the post. really helpful. can you maybe supplement it with some examples and figures depicting how it transforms the original data into seperable data ? $\endgroup$ Apr 17, 2018 at 6:05
  • 1
    $\begingroup$ @svk: I just formatted in a way it is understandable, I think hadi was the person who wrote the answer. If my guess is right, he might have copied from some book. Let's see if he replies back/suggests something. Else would write an answer $\endgroup$
    – Toros91
    Apr 17, 2018 at 6:08
  • 2
    $\begingroup$ as @Toros91 said, It's the combination of different sources, I highly recommend to see How to Win a Data Science Competition: Learn from Top Kagglers and also stata docs $\endgroup$ Apr 17, 2018 at 11:55
  • $\begingroup$ alrite. although can one of you please post some example code in python or matlab on how to transform x axes to sq.root or cube root scale. matlab has the loglog plot for the log scale. but for other scales, it'll be helpful to have a plot with these transformed axes $\endgroup$ Apr 19, 2018 at 4:47
  • $\begingroup$ Examples of the effect of the transformations can be found in this article: trainindata.medium.com/…. Examples and code can be found in this github repo: github.com/solegalli/… $\endgroup$
    – Sole G
    Jun 2, 2022 at 11:53
2
$\begingroup$

These specific ones could be pure heuristic. For images though it is pretty standard: change RGB to BGR and subtract the mean from every pixel. This is used in all contests/datasets like Imagenet, Pascal VOC, MS COCO. The reason is that the network is presented with a standardized dataset, as all images can be very different.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.