# Features selection in KNN

I have a naive question about using the K Nearest Neighbor algorithm: is feature selection more important in KNN than in other algorithms?

If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right? If I'm predicting height based on weight and age and gender, my model will get worse if I now add house numbers, because people will similar house numbers to me will be closer to me.

In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

Typically, an implementation of KNN will include the option to apply a weight. For example, in the package sklearn.neighbors.KNeighborsClassifier there is a parameter to supply weights. The trick will be trying to determine how to set the weight in your case. That said, supplying weights does not necessarily a offset the need to normalize the values.

• And if I select too many features, can I mess up my results? – Alex Dec 6 '18 at 7:39
• It’s not that too many will mess it up per se. If the features are sparsely populated, it just may not help much. For example, if you have a feature for full name, you would need to one-hot-encode it to use it in KNN. After encoding, there not a lot of commonality so, it does not contribute to the similarity score in a predictable fashion. In this example, drop the full name feature. Unless a high cardinality feature is known to be really important, it is probably not going to contribute in a meaningful way. – Skiddles Dec 6 '18 at 11:27
• Thanks for your responses. In my (ridiculous) house number example, wouldn’t adding house number cause some data points to now be closer than they otherwise would be? If someone has the same house number as me, they might be my nearest neighbor (no pun intended) even though house number has no predictive value for my height. Or maybe there’s something obvious that I’m missing? – Alex Dec 6 '18 at 11:53
• I think you get the picture. If you want to see how important your features are, one exercise you can do is train a ‘decision tree’. One nice function they provide is the ability to list the importance of the features to the algorithm. This may give you some unbiased feedback about the features. – Skiddles Dec 6 '18 at 12:34
• The weights param in KNeighborsClassifier is nothing to do feature selection. It determines if all of the k neighbours in the neighbourhood contribute equally, or if closer points influence the prediction more. – fordy May 13 at 16:37

KNN Algorithm does not provide any prediction for the importance or coefficients of variables. You might could apply another model like a regression (or a random-forest) to calculate the coefficients.

Otherwise, you could apply first some feature selection metrics (like Information Gain) and select the most informative features or apply weights consdidering the result of the metric. For the latter you could use a weighted euclidean distance for the finding the nearest neighbors of an instance or use the option of the weighted KNN in the scikit learn library in python.