When googling "weighted KNN", the results appear to be focused on weighting the nearest neighbor values after those neighbors have been determined. I'm looking for something that assigns a level of importance to various dimensions that could possibly change the neighbors that are considered nearest with the goal of maximizing the accuracy of predictions made using the resulting model.
For example, if I have a new observation defined as
[1, 2], #1 and #2 below would be considered the nearest neighbors assuming K=2 under a normal KNN (as I understand it):
However, if the dimensions were weighted by importance, such as
[0.1, 1], the items above would have respective distances of 0.502494, 0.50636, 0.360555 based on
sqrt((d1*w1)^2 + (d2*w2)^2), which would make #3 the overall nearest.
Given the goal of maximizing prediction accuracy, I'm wondering what methods are available to accomplish this? I'd like to potentially use this for both classification and regression - are there methods other than KNN that I should consider?