16
$\begingroup$

I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or log of continuous variable like loan_amount or applicant_income etc.

I just want to understand the reason behind it. Does it help improve our model prediction accuracy. Is it mandatory? or Is there any logic behind it?

Please provide some explanation if possible. Thank you.

$\endgroup$
26
$\begingroup$

This is done when the variables span several orders of magnitude. Income is a typical example: its distribution is "power law", meaning that the vast majority of incomes are small and very few are big.

This type of "fat tailed" distribution is studied in logarithmic scale because of the mathematical properties of the logarithm:

$$log(x^n)= n log(x)$$

which implies

$$log(10^4) = 4 * log(10)$$

and

$$log(10^3) = 3 * log(10)$$

which transforms a huge difference $$ 10^4 - 10^3 $$ in a smaller one $$ 4 - 3 $$ Making the values comparable.

$\endgroup$
  • 2
    $\begingroup$ Nice answer specially talking about exponential distributions. $\endgroup$ – Kasra Manshaei Oct 23 '18 at 13:40
  • 1
    $\begingroup$ @KasraManshaei I was speaking about power laws in particular (income being a typical example): extreme values in exponential distribution are by definition very rare. Therefore data which spans many orders of magnitude is usually power law. $\endgroup$ – Duccio Piovani Oct 23 '18 at 14:09
  • 1
    $\begingroup$ but of course in such cases log ---> ln, which absolutely doesnt change the point of the answer. $\endgroup$ – Duccio Piovani Oct 23 '18 at 14:10
  • $\begingroup$ Yes I got it. As you said not much changes. $\endgroup$ – Kasra Manshaei Oct 23 '18 at 14:37
6
$\begingroup$

Mostly because of skewed distribution. Logarithm naturally reduces the dynamic range of a variable so the differences are preserved while the scale is not that dramatically skewed. Imagine some people got 100,000,000 loan and some got 10000 and some 0. Any feature scaling will probably put 0 and 10000 so close to each other as the biggest number anyway pushes the boundary. Logarithm solves the issue.

$\endgroup$
  • $\begingroup$ Manshael, So I can use MinMaxScaler or StandardScaler right? or Is it necessary to take log? $\endgroup$ – Sai Kumar Oct 23 '18 at 13:26
  • $\begingroup$ Necessary. If you use scalers they compress small values dramatically. That's what I meant to say. $\endgroup$ – Kasra Manshaei Oct 23 '18 at 13:26
  • 2
    $\begingroup$ Yes. If you take values 1000,000,000 and 10000 and 0 into account. In many cases, the first one is too big to let others be seen properly by your model. But if you take logarithm you will have 9, 4 and 0 respectively. As you see the dynamic range is reduced while the differences are almost preserved. It comes from any exponential nature in your feature. In those cases you need logarithm as the other answer depicted. Hope it helped :) $\endgroup$ – Kasra Manshaei Oct 23 '18 at 13:30
  • 2
    $\begingroup$ Well, scaling! Imagine two variables with normal distribution (so there is no need for logarithm) but one of them in the scale of 10ish and the other in the scale of milions. Again feeding them to the model makes the small one invisible. In this case you use scalers to make their scales reasonable. $\endgroup$ – Kasra Manshaei Oct 23 '18 at 13:36
  • 1
    $\begingroup$ @KasraManshaei log(0) = -inf though. $\endgroup$ – JAD Oct 23 '18 at 14:01
5
$\begingroup$

In addition to the other answers, another side-effect of taking $\log{x}$ is that if $0 < x < \infty$, again for example with loans or incomes, basically anything that cannot become negative, the domain becomes $-\infty < \log{x} <\infty$.

This can be helpful, especially in return variables, if the model you are using is based on assuptions about the distribution of $x$. For example the assumption of normality in linear models.

$\endgroup$
3
$\begingroup$

Yet another reason why logarithmic transformations are useful comes into play for ratio data, due to the fact that log(A/B) = -log(B/A). If you plot a distribution of ratios on the raw scale, your points fall in the range (0, Inf). Any ratios less than 1 will be squished into a small area of the plot, and furthermore, the plot will look completely different if you flip the ratio to (B/A) instead of (A/B). If you do this on a logarithmic scale, the range is now (-Inf, +Inf), meaning ratios less than 1 and greater than 1 are more equally spread out. If you decide to flip the ratio, you simply flip the plot around 0, otherwise it looks exactly the same. On a log scale, it doesn't really matter if you show a ratio as 1/10 or 10/1, which is useful when there's not an obvious choice about which it should be.

$\endgroup$
3
$\begingroup$

You should look at the lognormal distribution.

People may use logs because they think it compresses the scale or something, but the principled use of logs is that you are working with data that has a lognormal distribution. This will tend to be things like salaries, housing prices, etc, where all values are positive and most are relatively modest, but some are very large.

If you can take the log of the data and it becomes normalish, then you can take advantage of many features of a normal distribution, like well-defined mean, standard deviation (and hence z-scores), symmetry, etc.

Similarly, addition of logs is the same as multiplication of the un-log'd values. Which means that you've turned a distribution where errors are additive into one where they're multiplicative (i.e. percentage-based). Since techniques like OLS regression require a normal error distribution, working with logs extends their applicability from additive to multiplicative processes.

$\endgroup$
  • 1
    $\begingroup$ If you want to compare items in a distribution-free way, wouldn't it be better to take percentiles or deciles and use those instead of the original value? $\endgroup$ – William Payne Oct 30 '18 at 13:04
  • 1
    $\begingroup$ @WilliamPayne Sure, you can use a distribution-free method, though you're also giving up some of the power of having a distribution... if your distributional assumptions are correct. With greater (correct) assumptions comes greater power. Percentiles are essentially ranks, so you throw away the distance information you have, and a particular sample's percentile is a point estimate. We'd generally prefer distributions to points. $\endgroup$ – Wayne Oct 19 at 12:22
0
$\begingroup$

I'd say the main reason is not distributional but rather because of the non linear relationship. Logs often capture saturating relationships...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.