How to appropriately set weights for weighted KNN

Question

I'm currently trying to predict a continuous variable using KNN. Instead of treating each neighbor equally I would like to use the weights to create a weighted average. The weights by themselves are not ideal, as the closer a neighbor the more I would like that neighbor to influence the final results.

This lead me to consider the inverse of each of the distances, but this doesn't handle the case where an instance is the exact same -> with a distance of 0.

Any recommendations on how to properly set the weights of each neighbor relative to their distance? Similar to how the inverse would handle this, but one that allows for 0 values.

Answer 1

Technically speaking the inverse of the distance should not pose any problem. Considering the case when you have to classify an observation (instance) whih is identical with one observation from the 'learned' data set, than the inverse distance is not defined by a direct formula since you have to divide by zero. But considering the math behind, inverse of the zero distance is positive infinity. Which means that, the weight for the identical point would dominate all other weights of the data instances. Thus the classification for that specific point could be taken as the class of the identical observation from the data set.

On the other hand the inverse of the euclidean distance is only one type of distance. You can always apply a kernel function on that to weight non-uniformly the contributions. For example you can choose a normal density with mean equals zero and variance equals 1. Than, your weight would be the value of density function for the euclidean distance.

Answer 2

A decaying exponential of the form $e^{-\alpha x}$ (where $x$ is the distance from the observation) is convenient in this situation. It has the nice feature that the weight is equal to $1$ when the observation lies exactly at one of your training points and decays to zero as $x\rightarrow \infty$.

What you need to decide is the scaling factor, $\alpha$, which can greatly affect your results, since the decaying exponential is nonlinear. If $\alpha$ is too small or large, then all $k$ points will be weighted nearly equally (assuming none of the $k$ points have a distance very close to zero). One approach is to pick a global value of $\alpha$ that is suitable for your data set. Another approach is to compute a new value of $\alpha$ for each set of $k$ neighbors under consideration, which will guarantee variability in the weights (as long as all $k$ neighbors are not equidistant).