7
$\begingroup$

I would like to train a binary classifier on feature vectors. One of the features is categorical feature with string, it is the zip codes of a country.

Typically, there is thousands of zip codes, and in my case they are strings. How can convert this feature into numerical?

I do not think that using one-hot-encoding is good as a solution for my case. Am I right by saying that? If yes, what would be a suitable solution?

Improve this question
$\endgroup$
10
  • $\begingroup$ Do you need the granularity of zip codes? Could you calculate/infer the distances from them? The reason I ask is that you may not have enough observations for some zip codes depending on the density of the data $\endgroup$
    – ImperativelyAblative
    Mar 3, 2016 at 20:31
  • $\begingroup$ No, the distance between zip codes is irrelevant in my case. $\endgroup$
    – Rami
    Mar 3, 2016 at 20:32
  • $\begingroup$ could you please elaborate the problem of your system and how the zip codes hold importance in the data? Are they helping in any inference that does not have a dead end? M $\endgroup$
    – RahulS
    Mar 3, 2016 at 21:12
  • $\begingroup$ yes, the zip code is an information that wight help in the classification task that I am trying to solve. $\endgroup$
    – Rami
    Mar 3, 2016 at 21:16
  • 1
    $\begingroup$ Welcome to the site @RahulS. Please leave comments such as this below the initial post, not as an "answer". $\endgroup$
    – Emre
    Mar 3, 2016 at 22:23

3 Answers 3

Reset to default
7
$\begingroup$

One-hot-encoded ZIP codes shouldn't present a problem with modern tools, where features can be much wider (millions, billions even), but if you really want you could aggregate area codes into regions, such as states. Of course, you should not use strings, but bit vectors. Two other dimensionality reduction options are MCA (PCA for categorical variables) and random projection.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ thanks, so you mean treating a binary vector of million (even billions) dimensions is not a problem for the actual tools (spark for example)? $\endgroup$
    – Rami
    Mar 3, 2016 at 21:12
  • 1
    $\begingroup$ Not in terms of the raw capacity to process it, but you have to be mindful of overfitting and the curse of dimensionality in such situations. If your training error significantly differs from your test error, use more regularization. $\endgroup$
    – Emre
    Mar 3, 2016 at 22:04
  • $\begingroup$ thanks Emre, ok so One-hot-encoder + dimensionality reduction is reasonable choice then. I also like what @RahulS suggested in applying TF-IDF to the column. I will try both and compare if I have time. thanks again $\endgroup$
    – Rami
    Mar 4, 2016 at 9:16
  • 2
    $\begingroup$ I wanted to add that while one-hot encoding zip will work just fine, a zip code is a content rich feature, which is ripe for value-added feature engineering. So you should think about the things it could add to your data if you inner join it to other zip code data sets. States can be extracted, latitude and longitudes can be extracted, average summer high temperatures, days per year of rain, gun ownership, socio-economics, upward mobility, average home values, average education level, degree of urbanization, ... The list goes on and on, so zip can be highly rich in adding information. $\endgroup$
    – AN6U5
    Jul 29, 2016 at 3:54
  • 1
    $\begingroup$ There are rather more effective algorithms than just OHE. For example, you can go with Empirical Bayes probablity per category of dependent target. Search for Daniele Barreca "High-cardinality categorical preprocessing" or entitiy embeddings arxiv.org/pdf/1604.06737.pdf $\endgroup$
    – Novitoll
    Oct 2, 2017 at 13:26
8
$\begingroup$

This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use frequency or the cdf of the frequency (to avoid noise generated by heavy-tailed pdf), and they achieved pretty good results with lightGBM. However, i cannot really explain why it works rigorously.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ Would mean encoding increase the risk of overfitting? Say, in reality the target label does not depend on the categorical feature, but by using mean encoding we introduce the artificial dependency of the target on the categorical feature. $\endgroup$
    – Michael Larionov
    Jul 15, 2018 at 20:54
  • 2
    $\begingroup$ @MichaelLarionov, good thought! I didn't think of it until you mention. The answer is NO! If the target label does not depend on a feature, then the distribution of this feature is independent of the target. So in asymptotic cases, the values of all mean encoded category feature are the same. For example, if a disease has nothing to do with age groups, then the infection rate will be the same across all age groups. And this does not affect most of the models. $\endgroup$
    – Diansheng
    Jul 16, 2018 at 3:18
3
$\begingroup$

You can use embedding which is mentioned in the comments. e.g. A general blog post, Keras documentation for embedding layer which can be used to learn the embedding. This is widely used by deep learning models when you need to reduce the number of features and it works for one categorical feature as well.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.