Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or log of continuous variable like loan_amount or applicant_income etc.
I just want to understand the reason behind it. Does it help improve our model prediction accuracy. Is it mandatory? or Is there any logic behind it?
Please provide some explanation if possible. Thank you.
This is done when the variables span several orders of magnitude. Income is a typical example: its distribution is "power law", meaning that the vast majority of incomes are small and very few are big.
This type of "fat tailed" distribution is studied in logarithmic scale because of the mathematical properties of the logarithm:
$$log(x^n)= n log(x)$$
which implies
$$log(10^4) = 4 * log(10)$$
and
$$log(10^3) = 3 * log(10)$$
which transforms a huge difference $$ 10^4 - 10^3 $$ in a smaller one $$ 4 - 3 $$ Making the values comparable.
Mostly because of skewed distribution. Logarithm naturally reduces the dynamic range of a variable so the differences are preserved while the scale is not that dramatically skewed. Imagine some people got 100,000,000 loan and some got 10000 and some 0. Any feature scaling will probably put 0 and 10000 so close to each other as the biggest number anyway pushes the boundary. Logarithm solves the issue.
In addition to the other answers, another side-effect of taking $\log{x}$ is that if $0 < x < \infty$, again for example with loans or incomes, basically anything that cannot become negative, the domain becomes $-\infty < \log{x} <\infty$.
This can be helpful, especially in return variables, if the model you are using is based on assuptions about the distribution of $x$. For example the assumption of normality in linear models.
You should look at the lognormal distribution.
People may use logs because they think it compresses the scale or something, but the principled use of logs is that you are working with data that has a lognormal distribution. This will tend to be things like salaries, housing prices, etc, where all values are positive and most are relatively modest, but some are very large.
If you can take the log of the data and it becomes normalish, then you can take advantage of many features of a normal distribution, like well-defined mean, standard deviation (and hence z-scores), symmetry, etc.
Similarly, addition of logs is the same as multiplication of the un-log'd values. Which means that you've turned a distribution where errors are additive into one where they're multiplicative (i.e. percentage-based). Since techniques like OLS regression require a normal error distribution, working with logs extends their applicability from additive to multiplicative processes.
Yet another reason why logarithmic transformations are useful comes into play for ratio data, due to the fact that log(A/B) = -log(B/A). If you plot a distribution of ratios on the raw scale, your points fall in the range (0, Inf). Any ratios less than 1 will be squished into a small area of the plot, and furthermore, the plot will look completely different if you flip the ratio to (B/A) instead of (A/B). If you do this on a logarithmic scale, the range is now (-Inf, +Inf), meaning ratios less than 1 and greater than 1 are more equally spread out. If you decide to flip the ratio, you simply flip the plot around 0, otherwise it looks exactly the same. On a log scale, it doesn't really matter if you show a ratio as 1/10 or 10/1, which is useful when there's not an obvious choice about which it should be.
I'd say the main reason is not distributional but rather because of the non linear relationship. Logs often capture saturating relationships...
