1
$\begingroup$

I want to understand the assumptions made by supervised machine learning models.

I've heard it said many times that 'you need to make sure your feature variables are normally distributed for your ML models to work.' However, when I looked up the assumptions for Linear Regression, I found many conflicting viewpoints.

This post and this post mention normality of error distributions. The latter even says that features do not need to be normally distributed - just the errors.

Statistics solutions says that features do need multivariate normality, as do a lot of top beginner ML courses such as Machine Learning A-Z.

Wikipedia says that features do not need to be normally distributed, but it has a major influence on the precision of estimates.

Which algorithms assume that the feature variables are normally distributed? For ones that don't, why is it beneficial to scale your features so that they are more normally distributed?

$\endgroup$
1
$\begingroup$

If you are distinguishing between Statistics and Machine Learning, then you need to define boundary between the two, and that boundary is going to be opinion-based. It is a matter of definition which algorithms fall into Machine Learning and which under Statistics.

I'll try to give some examples without committing to one or the other.

  1. Linear regression expects the errors (residuals) to be normally distributed. This comes from the maximum likelihood and the $x^2$ term in the normal distribution's PDF.
  2. Logistic regression expects two normally distributed classes, with equal variance. This comes from the Bayesian probability of class membership being a logistic function iff the classes are normal.
  3. Linear discriminant analysis expects two normal-multivariate distributed classes with the same covariance matrix.

You can apply these algorithms even if your data violate the assumption, but then the results (in statistics: parameter estimates) won't be (quite) correct. How much they'll deviate from the real values depends on the extent of the assumption violation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.