kNN is a distance-based method, so it requires the input to be in numerical form.

I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color).

Since the input has to be in numerical form, we have to encode the color feature before applying the kNN imputer.

Using ordinal encoding doesn't seem like a good idea. If we assign numbers 1-10 to colors, then the distance-based measurement will assume that the distance between color 1 and 3 is the same as the distance between color 2 and 4. In reality no such relationship exists.

We could use one-hot encoding. Let's say we end up with 10 one-hot encoded color columns, including a column which indicates the missing values. Our task would then be to apply the imputation to the rows which have a 1 in the one-hot-encoded missing-color column. And we would want the imputation to decide which of the other 9 one-hot color columns would have a 1 instead of a 0. But this is generally not at all how kNN imputation works.

Can you advise if using kNN imputation in this case is possible? How can I apply it?

  • 1
    $\begingroup$ yes by using majority voting even categorical features can be used with kNNs. (Always take care to resolve ties) $\endgroup$
    – Nikos M.
    Mar 29 at 16:51
  • $\begingroup$ @NikosM. Can you elaborate? As far as I know kNN finds x most similar datapoints to the one you are trying to impute. Then the weighted average of these points is taken. How can you apply that to a ordinal-encoded or one-hot-encoded feature? Where would you apply majority voting? $\endgroup$ Mar 30 at 17:17
  • 1
    $\begingroup$ For majority voting the point is assigned the feature that most neighbors share (majority), with care to resolve ties. This works for categorical variables as well. For example kNN for colors if most neighbors have blue then blue it is. Average of neighbors is another option, equally valid to majority voting, but does not work with categorical variables $\endgroup$
    – Nikos M.
    Mar 30 at 17:25
  • $\begingroup$ Thank you. Do you know any python package that implements majority voting? $\endgroup$ Mar 30 at 17:28
  • $\begingroup$ kNN with majority voting is a very simple algorithm to implement yourself. For example: kdnuggets.com/2016/01/…, stats.stackexchange.com/questions/45580/…, stats.stackexchange.com/questions/275872/… $\endgroup$
    – Nikos M.
    Mar 30 at 17:35

K-nearest neighbors algorithm (k-NN) can work with categorical features using counts as a distance metric. The nearest neighbors would have the closest count frequency.

Also, color is ordinal. Each color is a specific frequency on the visible light spectrum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.