Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have more than 15 unique values such as country.
Do I still use one-hot encoding? Or is there an alternative? Is using regression on this dataset a good idea?
While most answers here suggest to use various encoding schemes, I would like to propose a different approach: collapsing categories. The idea is that if there are two (or more) similar categories, you can unite the, into a single category, thus reducing the dimensionality of the feature/variable. Also, if there are some categories with expected low frequencies, you may collapse them into as "other" category.
If you have high cardinality categorical data(+10 distinct values) you can do Target Encoding.
One hot Encoding in high cardinality scenarios has the following drawbacks:
The input data for the model becomes very wide, and neither an optimal nor an efficient approach is guaranteed.
The created features become sparse(most of the levels hardly appear in the data)
One Hot Encoding does not handle new or unseen categories
One option here is to do target encoding.
Here is an intro. Find here a python package with several implementations.
Target Encoding benefits:
Drawbacks
For a more comprehensive literature review:
[edit] See also Carlos' answer, I think it's better than mine.
You should use one hot encoding for the categorical features. Replacing categorical values with numerical ones would be a bad idea, because it introduces order between the values and the model would try to find patterns based on this order (e.g. 'x < 4').
If there were really too many different values then it's often a good idea to remove or replace the least common ones, but it doesn't seem to be an issue with this data.
For the record, more than 15 different values is nothing to be afraid of. For example when working with text data it's common to work with thousands of values for a word.
You are right that most of the algorithms can digest only numerical data, i.e. the categorical features need to be converted to the numerical ones before running the regression.
Besides straightforward one-hot encoding and already mentioned target encoding you could try the following approaches
Collapse the less frequent categories into one bucket (e.g. "other") and then apply the one-hot encoding. In particular, this approach might work fine for the variable "country" in your dataset: 7 most frequent categories / countries cover more than 95% of observations.
Use some proxy features, i.e. instead of using the country name directly take some relevant statistics for this country. These would be problem specific, but you could start with something like population, area, population density, income per capita, GDP, GDP per capita, statistics about education levels.
There are a number good options for encoding categorical data with high cardinality. If using Python, the Category Encoders package has like a dozen options as of this writing. I wrote a guide to using some of them here.
Category Encoders includes several Bayesian encoders, including Target encoder.
Binary and Hashing encoders help reduce dimensionality, but might make it a bit harder for your model to pick up signal in your data.
