3
$\begingroup$

My professor said that the "holy grail of regression" is the function E(Y|X=x) i.e. the conditional expectation of Y on X. In practice, you'd take a small window of X and take the average value of Y for all observations that lie in the window.

The professor said that this is basically the best prediction you can make, but we don't usually do it because the curse of dimensionality reduces its effectiveness in when # of predictors is large. So it seems that local averaging (KNN regression is a type of this) is good with few predictors. However, in most articles and stats classes, I always see linear regression being used even in low dimensions. Why isn't local averaging used more often?

$\endgroup$
1
$\begingroup$

Local averaging regression is much more complicated model than a simple linear regression as it's a mixture of many individual smaller models. Local regression is actually quite popular in the literature, and has been tested extensively.

Local regression models require large dense sampling data set, which is not common for a statistical student. If you don't have complicated relationships to model, you probably don't need it.

$\endgroup$
0
$\begingroup$

In practical terms, local averaging would utilize more amount of memory, especially when the number of predictors soars, making test-set predictions slow. That's the major caveat of a KNN regressor. Hope it helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.