I've been reading about K-nearest neighbors algorithm and want to clarify few things.

If we have 2 features we could simply plot it on 2-d plane and calculate distance by using euclidean distance or Manhattan distance.

When there are more than 3 features, exceeding 3-d going into 4-d and more I've read that we use PCA to reduce dimension to 2-D and then calculate distance on PCA plot.

But my question is, is that only way? so in order to use KNN for more than 3 features we must use PCA?


No, you can definitely search for k-NN with more than 2-dimension data. Here is an example based on sklearn:

X = [[0, 0, 0], [3, 3, 3], [1, 2, 3]]
from sklearn.neighbors import NearestNeighbors
neigh = NearestNeighbors(n_neighbors=2)
print( neigh.kneighbors([[2,2,2]]) )

PCA is used to reduce the input dimensionality but this is not mandatory before searching k nearest neighbors (it is often used in tutorials so the data could be visualized on a 2-d plot).

One thing to know/understand about k-NN is that if you plan to use it for classification, it will handle features with a lot of information the same way as the features with no information (if you normalize them). PCA could be used to handle this problem (but this is not the only way and would not always work, but I think this is another question :) ).

  • $\begingroup$ Thanks for your answer. If dimension exceed 3-d how does knn measure distance between two data points? In 2-D it uses either Manhattan or Euclidean which makes sense but what distance for 4-D, 5-D and so on? $\endgroup$ – haneulkim Aug 2 '20 at 23:56
  • 1
    $\begingroup$ Euclidean and Manhattan work in higher dimension. See the corresponding definition in wikipedia of euclidean and manhattan for the formula. $\endgroup$ – etiennedm Aug 3 '20 at 7:03
  • $\begingroup$ Thanks! So point of PCA is simply to visualize data points of high dimensions? $\endgroup$ – haneulkim Aug 3 '20 at 23:53
  • $\begingroup$ No, it is first a tool for dimensionality reduction which is used for several reasons (depending on the context). You'll find more info about dimensionality reduction here. Regarding PCA, you can find tutorials that explain it like this one. Please feel free to ask another question if something remains unclear. $\endgroup$ – etiennedm Aug 4 '20 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.