6
$\begingroup$

I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to log(SalePrice) see below:

enter image description here

enter image description here

I can see that taking a log transform reduced the skewness of data and made it more normal-like. But, i was wondering will it improve my model's performance or is use-full in any way. If it is then how a normal distribution of target variable was a catalyst?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter?

Transforming your target via a logarithmic function linearizes your target. Which is useful for many models which expect linear targets. Scikit-Learn has a page describing this phenomenon: https://scikit-learn.org/stable/auto_examples/compose/plot_transformed_target.html

Important to note

If you modify your targets before training, you should apply the inverse transform at the end of your model to compute your "final" prediction. That way, your performance metrics can be comparable.

Intuitively, imagine that you have a very naive model which returns the average target regardless of the input. If your targets are skewed, it means that you will under-/over-shoot for a majority of the predictions. Because of this, the range of your error will be greater, which worsens scores such as the Mean Absolute or Relative Error (MAE/MSE). By normalizing your targets, you reduce the range of your error, which ultimately should improve your model directly.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Scaling the observations to get a smaller number as the standard deviation is cheating. You’re just changing the units. A standard deviation of 100cm is the same as a standard deviation of 1m, even though $1<100$ as raw numbers. $\endgroup$
    – Dave
    Commented May 25, 2020 at 13:41
  • $\begingroup$ @Dave I fully agree that the standard deviation doesn't change, only the unit. But here the transformation mostly modifies the skewness of the distribution. Also, I don't agree with you calling this "cheating". There are plenty of preprocessing steps which normalize, standardize inputs, which are perfectly valid. $\endgroup$
    – Valentin Calomme
    Commented May 25, 2020 at 14:03
  • $\begingroup$ The cheating is to get data with a variance of 100, divide through (or otherwise transform) to get a variance of 1, and run off to your boss to tell her about how much you’ve reduced the variability. And look at my comment to Kasra’s post for my thoughts about the pooled distribution of the response variable. $\endgroup$
    – Dave
    Commented May 25, 2020 at 14:07
  • $\begingroup$ I agree with your comment but I'm still confused about why it pertains to the question or my answer. The goal has nothing to do with modifying the variance (or not). The question is why preprocessing the targets with a logarithmic function generally improves the performance of an ML model. $\endgroup$
    – Valentin Calomme
    Commented May 25, 2020 at 14:27
  • $\begingroup$ Maybe you get better performance; maybe you don’t. However, by doing such a transformation, you’re changing the problem, so a smaller MSE after doing the transformation is not on its own evidence of superior model performance. Of course you have smaller MSE when you compress the data and work with smaller numbers, but a cell biologist gets lower MSE when she makes her measurements in light years instead of nanometers. $\endgroup$
    – Dave
    Commented May 25, 2020 at 14:35
3
$\begingroup$

Well ... there are many aspects from which one can answer this question (Like Valentin's answer ... +1!) as Machine Learning and Data Mining are very much about distributions in general. I just mention a few that come to my mind first.

  • Some models assume Gaussian distribution e.g. K-means. Imagine you want to apply K-means on this data without log-transform. K-means will have much difficulties with the original feature but the log-transform makes it pretty proper for K-means as "mean" is a better representative of samples in a Gaussian distribution rather than skewed ones.
  • Some statistical analysis techniques also assume gaussianity. ANOVA works the best with (and actually designed for) normally distributed data (specially in small sample populations). The reason is simply the fact that it is mainly dealing with mean and sample variance to determine the "center" and "variation" of sample populations and it makes sense the most in Gaussian distribution.
  • All in all, computation with adjusted features is more robust than skewed features. Skewed features have uneven range which is an issue specially if their range is huge (like your example). Adjusted (engineered) features are not going to necessarily become Gaussian but having a smaller and more even range for example.
Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ The Gaussian assumption in ANOVA and all other OLS regressions is about the error term (estimated by the residuals), not about the pooled distribution of all observations of the response variable. Further, this assumption is for doing inference, not prediction. Inference is the goal in ANOVA, but machine learning is about prediction. Parameter confidence intervals and F-testing nested models is less important. $\endgroup$
    – Dave
    Commented May 25, 2020 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.