3
$\begingroup$

I'm building a model to predict the lifetime value of a client based on the relational data we have on them. The user table has a bunch of one-to-many child tables that might be predictive. Grossly simplified, the child features boil down to things like:

  • a list of item categories that they've bought in the past
  • a list of the predominant colors in ads they've clicked on
  • etc, etc

In each case, the obvious feature comprises a list of ~ 0-10 choices from a categorical variable. I have several of these features, some of which have as many as ~10k discrete values, so one-hot encoding would get very wide, very fast.

Aside: if there a term of art for this kind of "list-of-tags feature" that I'm referring to as "choose many categorical", please tell me.

Question: Is there an dense encoding scheme that works with choose-many categorical features?

Improve this question
$\endgroup$
2
  • $\begingroup$ I'd suggest you do some feature engineering and see if you can merge some tags together to reduce the number. If not, working with inputs of 10k+ columns is not impossible with sparse matrix e.g. docs.scipy.org/doc/scipy/reference/sparse.html. $\endgroup$
    – Jean-François Savard
    Nov 21, 2020 at 1:59
  • $\begingroup$ Have you considered label-encoding or multi-hot encoding? It reduces the size of your embedding compared to one-hot-encoding. I made a post about the differences between them with examples: stats.stackexchange.com/a/467672/264183 $\endgroup$
    – Tinu
    Nov 21, 2020 at 11:13

2 Answers 2

Reset to default
4
$\begingroup$

If your algorithm is based on gradient descent optimization, you can use embeddings, which are dense representation spaces for discrete elements.

Embeddings are supported by most deep learning frameworks such as pytorch or tensorflow.

Update: the fact that you want to have multiple discrete values does not prevent the possibility of using embeddings: you can just add all vectors together in a single value. The most straightforward approach for this would be to have a constant length for the list (equal to the maximum number of elements in all lists, or a sensible maximum value), filling with "padding" items the positions that are not needed. If you want to take the sequential appearance of the elements into account, instead of adding the vectors together you could apply convolutional layers or an LSTM over the embedded vectors.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This post is specifically asking about "choose-many" -- a feature where each value is a list of choices. Can you point me to any reading that addresses this case, rather than trivial single-choice categorical inputs? $\endgroup$
    – Autumn
    Nov 23, 2020 at 17:57
  • 1
    $\begingroup$ Sorry, I had not noticed that. I have updated my answer with more details to use embeddings in this scenario. $\endgroup$
    – noe
    Nov 23, 2020 at 18:29
  • $\begingroup$ That seems unintuitive, but plausible! I've upvoted, will be back to accept the answer after I've tried to apply it. $\endgroup$
    – Autumn
    Nov 23, 2020 at 19:42
2
$\begingroup$

There are many ways to encode categorical features in the category encoders library you can find many of them.

The one that seems more promising given your data is target encodinc

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The example you've linked describes using target encoding for single choices. I'm asking about multiple choices. EG: row 0 has an "animal" feature of [cat, hamster]. Row 1 is [cat]. Row 2 is [cat, dog, hamster]. Row 3 is [dog, cat]. Etc. Many of the values will be a unique list, so target encoding could result in a very sparse probability vector for most entries. $\endgroup$
    – Autumn
    Nov 23, 2020 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.