0
$\begingroup$

When googling "weighted KNN", the results appear to be focused on weighting the nearest neighbor values after those neighbors have been determined. I'm looking for something that assigns a level of importance to various dimensions that could possibly change the neighbors that are considered nearest with the goal of maximizing the accuracy of predictions made using the resulting model.

For example, if I have a new observation defined as [1, 2], #1 and #2 below would be considered the nearest neighbors assuming K=2 under a normal KNN (as I understand it):

  1. [1.5, 2.5] d=0.707107
  2. [1.8, 1.5] d=0.943398
  3. [4.0, 2.2] d=3.006659

However, if the dimensions were weighted by importance, such as [0.1, 1], the items above would have respective distances of 0.502494, 0.50636, 0.360555 based on sqrt((d1*w1)^2 + (d2*w2)^2), which would make #3 the overall nearest.

Given the goal of maximizing prediction accuracy, I'm wondering what methods are available to accomplish this? I'd like to potentially use this for both classification and regression - are there methods other than KNN that I should consider?

$\endgroup$
0
$\begingroup$

You can modify the matrix used to compute the distances between points. By default, KNN with quantitative features is used with Euclidean distances, this is, the identity matrix in the equation (x-y)' I (x-y) (it returns the squared Euclidean distance between point X and Y).

Instead of the identity matrix, you can use other positive definite matrices and compute Mahalanobis distances.

Other modifications you can do is manipulate each feature. This also affects the output of KNN, aside of standardizing, you can try non linear modifications of the features, reducing kurtosis or skewness, for example.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Is there a structured way to do this that will maximize prediction accuracy? $\endgroup$ – SuperCodeBrah Apr 26 at 17:34
  • $\begingroup$ Not sure in your case, but I suggest you can try Discriminant Adaptive Nearest Neighbor Classification (DANN). $\endgroup$ – javierazcoiti Apr 30 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.