3
$\begingroup$

I have a naive question about using the K Nearest Neighbor algorithm: is feature selection more important in KNN than in other algorithms?

If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right? If I'm predicting height based on weight and age and gender, my model will get worse if I now add house numbers, because people will similar house numbers to me will be closer to me.

In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

I'll go through your questions one by one:

is feature selection more important in KNN than in other algorithms?

I don't think it is more important for kNN than for other kinds of algorithms.

If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right?

Correct. Neural Networks are "smarter" algorithms, they have internal weights that adjust to minimize a cost function. Less important features will be attributed comparatively lower importance with respect to highly predictive variables. This doesn't happen in kNN, in which prediction is based exclusively on distance between datapoints - and no information about relative importance of variables can be deduced from it.

In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

sklearn allows to manipulate kNN weights. But this weights distribution is not endogenous to the model (such as for Neural Networks, that learn that autonomously) but exogenous, i.e. you have to specify them, or find some methodology to attribute these weights a priori, before running your kNN algorithm. If you can do that, and you have good methodological reasons for it, then changing variables' weights can improve your model, but I'd be careful about it (you can easily overfit).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your response. So it seems like with kNN, one of my responsibilities is to have an idea of which features are predictive, so that I can set the weights and/or leave out the irrelevant ones. If I don’t do that, my model won’t be very good. Whereas with other learning algorithms, I can through all the features I’ve got at the system, and it will sort things out on its own - it will know to weight or ignore features as needed. Right? $\endgroup$
    – Alex
    Commented Jan 29, 2020 at 15:37
  • $\begingroup$ Neural Networks have the power to learn autonomously to ignore less relevant information. Apart from them, it's exactly as you wrote. And yes, with kNN it is your responsibility to chose a weights distribution if you need. $\endgroup$
    – Leevo
    Commented Jan 29, 2020 at 16:03
  • $\begingroup$ A simple regression would also learn to set the weight to a very low value, for an irrelevant input? $\endgroup$
    – Alex
    Commented Jan 29, 2020 at 16:51
  • $\begingroup$ In case of regressions it's a little bit different. Regressions' parameters represent the intensity of the association with the dependent variable. Its relevance could be captured by the associated p-values. This could be used as importance for some tasks, but I would be cautious. It all boils down to what you mean by "relevance". The best ML models for importance scores are, IMHO, Tree-based models such as Random Forests and XGBoost. Their mathematical structure allows you to calculate "importance scores", literally. $\endgroup$
    – Leevo
    Commented Jan 29, 2020 at 17:21
0
$\begingroup$

Typically, an implementation of KNN will include the option to apply a weight. For example, in the package sklearn.neighbors.KNeighborsClassifier there is a parameter to supply weights. The trick will be trying to determine how to set the weight in your case. That said, supplying weights does not necessarily a offset the need to normalize the values.

Improve this answer
$\endgroup$
5
  • $\begingroup$ And if I select too many features, can I mess up my results? $\endgroup$
    – Alex
    Commented Dec 6, 2018 at 7:39
  • $\begingroup$ It’s not that too many will mess it up per se. If the features are sparsely populated, it just may not help much. For example, if you have a feature for full name, you would need to one-hot-encode it to use it in KNN. After encoding, there not a lot of commonality so, it does not contribute to the similarity score in a predictable fashion. In this example, drop the full name feature. Unless a high cardinality feature is known to be really important, it is probably not going to contribute in a meaningful way. $\endgroup$
    – Skiddles
    Commented Dec 6, 2018 at 11:27
  • $\begingroup$ Thanks for your responses. In my (ridiculous) house number example, wouldn’t adding house number cause some data points to now be closer than they otherwise would be? If someone has the same house number as me, they might be my nearest neighbor (no pun intended) even though house number has no predictive value for my height. Or maybe there’s something obvious that I’m missing? $\endgroup$
    – Alex
    Commented Dec 6, 2018 at 11:53
  • $\begingroup$ I think you get the picture. If you want to see how important your features are, one exercise you can do is train a ‘decision tree’. One nice function they provide is the ability to list the importance of the features to the algorithm. This may give you some unbiased feedback about the features. $\endgroup$
    – Skiddles
    Commented Dec 6, 2018 at 12:34
  • $\begingroup$ The weights param in KNeighborsClassifier is nothing to do feature selection. It determines if all of the k neighbours in the neighbourhood contribute equally, or if closer points influence the prediction more. $\endgroup$
    – fordy
    Commented May 13, 2019 at 16:37
0
$\begingroup$

KNN Algorithm does not provide any prediction for the importance or coefficients of variables. You might could apply another model like a regression (or a random-forest) to calculate the coefficients.

Otherwise, you could apply first some feature selection metrics (like Information Gain) and select the most informative features or apply weights consdidering the result of the metric. For the latter you could use a weighted euclidean distance for the finding the nearest neighbors of an instance or use the option of the weighted KNN in the scikit learn library in python.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.