8
$\begingroup$

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have more than 15 unique values such as country.

Do I still use one-hot encoding? Or is there an alternative? Is using regression on this dataset a good idea?

$\endgroup$

5 Answers 5

5
$\begingroup$

While most answers here suggest to use various encoding schemes, I would like to propose a different approach: collapsing categories. The idea is that if there are two (or more) similar categories, you can unite the, into a single category, thus reducing the dimensionality of the feature/variable. Also, if there are some categories with expected low frequencies, you may collapse them into as "other" category.

$\endgroup$
5
$\begingroup$

If you have high cardinality categorical data(+10 distinct values) you can do Target Encoding.

One hot Encoding in high cardinality scenarios has the following drawbacks:

  • The input data for the model becomes very wide, and neither an optimal nor an efficient approach is guaranteed.

  • The created features become sparse(most of the levels hardly appear in the data)

  • One Hot Encoding does not handle new or unseen categories

One option here is to do target encoding.

Here is an intro. Find here a python package with several implementations.

Target Encoding benefits:

  • High cardinality problem is handled
  • Categories are ordered allowing for easy extraction of the information and model simplification

Drawbacks

  • Overfitting

For a more comprehensive literature review:

  1. Quantile Encoder: Tackling High Cardinality Categorical Features in Regression Problems
  2. Fairness implications of encoding protected categorical attributes
$\endgroup$
4
$\begingroup$

[edit] See also Carlos' answer, I think it's better than mine.

You should use one hot encoding for the categorical features. Replacing categorical values with numerical ones would be a bad idea, because it introduces order between the values and the model would try to find patterns based on this order (e.g. 'x < 4').

If there were really too many different values then it's often a good idea to remove or replace the least common ones, but it doesn't seem to be an issue with this data.

For the record, more than 15 different values is nothing to be afraid of. For example when working with text data it's common to work with thousands of values for a word.

$\endgroup$
5
  • $\begingroup$ Would not Target Encoding will be helpful here? Its a way to deal with high cardinality categorical features and have the features ordered in a meaningful way $\endgroup$ Dec 25, 2020 at 10:13
  • $\begingroup$ @CarlosMougan this alternatve didn't even cross my mind. so thank you very much for your comment! I certainly make mistakes from time to time and it's really helpful when somebody points it out. Do you want to write an answer yourself or I just fix mine? $\endgroup$
    – Erwan
    Dec 25, 2020 at 11:32
  • $\begingroup$ I turned the countries column into a continent column so now there are only 4 different values. Is this a good approach? $\endgroup$
    – Cinemato
    Dec 25, 2020 at 13:17
  • $\begingroup$ @Erwan no problem!! Happens often :) I hope you agree with mine :) $\endgroup$ Dec 26, 2020 at 8:10
  • 1
    $\begingroup$ @Cinemato unless you have a good reason to do so, I don't think it's a good idea to simplify the data this way since you're losing potentially valuable information. Usually with OHE what I do is this: observe the distribution of the values, and remove only the least frequent values (or replace them with a 'default' special value). In any case it's always worth trying different options by observing the performance on a validation set. $\endgroup$
    – Erwan
    Dec 26, 2020 at 10:40
3
$\begingroup$

You are right that most of the algorithms can digest only numerical data, i.e. the categorical features need to be converted to the numerical ones before running the regression.

Besides straightforward one-hot encoding and already mentioned target encoding you could try the following approaches

  1. Collapse the less frequent categories into one bucket (e.g. "other") and then apply the one-hot encoding. In particular, this approach might work fine for the variable "country" in your dataset: 7 most frequent categories / countries cover more than 95% of observations.

  2. Use some proxy features, i.e. instead of using the country name directly take some relevant statistics for this country. These would be problem specific, but you could start with something like population, area, population density, income per capita, GDP, GDP per capita, statistics about education levels.

$\endgroup$
1
$\begingroup$

There are a number good options for encoding categorical data with high cardinality. If using Python, the Category Encoders package has like a dozen options as of this writing. I wrote a guide to using some of them here.

Category Encoders includes several Bayesian encoders, including Target encoder.

Binary and Hashing encoders help reduce dimensionality, but might make it a bit harder for your model to pick up signal in your data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.