# How to work with Log-transformation?

I'm beginning my data science journey and I've faced a challenge that confuses me a bit. I have a set with few features and a target variable whose raw distribution is highly skewed. I've read that it's possible to use a log transformation to normalize the target variable (loss in $) and thus increase the accuracy. When I train my model with "y_raw", using MAE I get an error of 306k. When I log-transform y = y.transform(np.log) I get MAE accuracy of around 2 (log-transformed units I suppose?), which is e^2 = 7.39 (y_raw). This is a significant drop from 306k to only 7.39 ($) (or am I getting it wrong?), so I am a bit suspicious about it.

So here are my questions: 1) Did I get it correct that the error rate drop from 306k to only 7.39 is real and is valid? 2) How do I make a predictions from there? If I feed a sample to my model, receive a log-transformed output, lets say it returned a prediction of y_log = 10. Do I then simply use an inverse of it by placing e^10 = 22,026.5 and will it be my final prediction?

Taking the log doesn't result in a normally-distributed target; it would tend to if the target was log-normally distributed, and you have something normalish there, not quite. But, this distribution isn't actually what matters.

What taking the log does is change your model of how errors arise when fitting a regressor. You're now saying that the target values are $$e^{P + \epsilon}$$ where $$P$$ is your model's prediction and $$\epsilon$$ is Gaussian noise. Or: $$e^P e^\epsilon$$. That part directly interacts with the assumptions in your regressor.

So what you're finding is that on average the predictions are wrong by a factor of 7.39, not +/- $7.39. What you really want to do is evaluate MAE on actual target values vs $$e^P$$ . You probably have a better model but not that much better. • So what you're finding is that on average the predictions are wrong by a factor of 7.39.- How can I easily transform it to the "raw" numbers to estimate the error rate in terms of$? By using my log-MAE result? Jun 7, 2020 at 16:17
• You need to exponentiate your predictions to get them in dollars, then compare the absolute differences from actual values. Jun 7, 2020 at 23:18
• I'm a little bit confused now. By using the obtained error coefficient, namely 2, isn't e^2 = 7.39 $follows the way you described? Jun 8, 2020 at 11:54 • If your log-space prediction is$P$and error is$\epsilon$in log space, then your prediction in dollars is$e^P$but the error in dollars is not$e^\epsilon$. It's$e^{P+\epsilon} - e^P = e^P(e^\epsilon - 1)\$. It's multiplicative in dollars. Jun 8, 2020 at 14:55
• Got it! It is clear now. Thank you! Jun 9, 2020 at 10:40

Of course your error rate is going to decrease. Remember that your changes in MAE values may come from the fact that the scales of your original variable and that variable transformed by a logarithm ARE NOT THE SAME, and mean is scale-dependant.