I have a naive question about using the K Nearest Neighbor algorithm: is feature selection more important in KNN than in other algorithms?

If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right? If I'm predicting height based on weight and age and gender, my model will get worse if I now add house numbers, because people will similar house numbers to me will be closer to me.

In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

Typically, an implementation of KNN will include the option to apply a weight. For example, in the package sklearn.neighbors.KNeighborsClassifier there is a parameter to supply weights. The trick will be trying to determine how to set the weight in your case. That said, supplying weights does not necessarily a offset the need to normalize the values.

  • And if I select too many features, can I mess up my results? – Alex Dec 6 at 7:39
  • It’s not that too many will mess it up per se. If the features are sparsely populated, it just may not help much. For example, if you have a feature for full name, you would need to one-hot-encode it to use it in KNN. After encoding, there not a lot of commonality so, it does not contribute to the similarity score in a predictable fashion. In this example, drop the full name feature. Unless a high cardinality feature is known to be really important, it is probably not going to contribute in a meaningful way. – Skiddles Dec 6 at 11:27
  • Thanks for your responses. In my (ridiculous) house number example, wouldn’t adding house number cause some data points to now be closer than they otherwise would be? If someone has the same house number as me, they might be my nearest neighbor (no pun intended) even though house number has no predictive value for my height. Or maybe there’s something obvious that I’m missing? – Alex Dec 6 at 11:53
  • I think you get the picture. If you want to see how important your features are, one exercise you can do is train a ‘decision tree’. One nice function they provide is the ability to list the importance of the features to the algorithm. This may give you some unbiased feedback about the features. – Skiddles Dec 6 at 12:34

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.