1
$\begingroup$

I have a dataset of 1600 rows and 28 columns. Only one column is partially complete with 1300 records. The rest is NaN. I did a value count of this columns and it has 84 different categories that are nominal. What is the best way to impute this column. I need to convert these in numbers impute it and then convert back. I understand that One-Hot encoding does not work in this case because of the high cardinality.

What is the best way to approach this problem?

$\endgroup$

2 Answers 2

1
$\begingroup$

You may want to look into Target Encoding as an example:

https://contrib.scikit-learn.org/category_encoders/targetencoder.html

Another post from the forums:

Problem with converting string to dummy variables

$\endgroup$
2
  • $\begingroup$ as I understand from the first link target encoder is for binomial or continuous. unfortunately my data in that column is not either of them $\endgroup$
    – user105068
    Sep 21, 2020 at 22:18
  • $\begingroup$ Hi again @erialdo! The terminology is confusing! In the case of target encoding, the "target" is not the original data but what it is converted into. So, it will take categorical data and convert it to either a binomial or continuous target. Another link may help with seeing the details: maxhalford.github.io/blog/target-encoding $\endgroup$ Sep 22, 2020 at 0:04
1
$\begingroup$

I will make a biased suggestion, reading this paper might provide some insights

"Quantile encoder: tackling high cardinality categorical features in regression problems"

Even thought there are many methods, with many of them implemented in the category encoders package (https://contrib.scikit-learn.org/category_encoders/). This paper can serve as good understanding.

In case you are dealing with socially sensitive data, you might want to have a look at this paper "Fairness implications of encoding protected categorical attributes". That you can find it on:

I hope it helps :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.