My data contains multiple observations of categorical feature.The feature space is medical symptoms, so the data for this feature is like : ['fever','pain','yellow skin' .... ] .The amount of symptoms observations per sample is not fixed and i have around 50 different symptoms

how can i encode this feature into something that ML model can deal with ? the order of the symptoms in the array is not important.

i tried one hot encoding but projecting feature space of 50 category levels into 50 indicators means losing information (having a sparse matrix )

any ideas?

  • $\begingroup$ Depends what model you are selecting? e.g. NN or other models like Decision Trees? $\endgroup$ Jan 14 '19 at 5:41
  • $\begingroup$ @MajidMortazavi would like to test it on both NN and some other tree-based methods such as boosting algorithms . for both of the types i'm not sure i know how to project my features since non of them can handle feature with multiple values $\endgroup$
    – Latent
    Jan 14 '19 at 8:03

What you need is Encoding Categorical variables. This topic is discussed in tons of blogposts, it is definitely worth checking out this recent article that nicely and extensively go through most of the methods, since I do not intent to rewrite again what is out there but rather giving you my personal experience.

I asked you earlier what algorithms you are after, simply that could change your choice of encoding method. Some encoding methods like One-hot-encoding makes your feature space very sparse when your categorical variable is very cardinal (usually not recommended!), and it is best to go with sparse-aware algorithms.

In a nutshell, I suggest to start with the following (classic and simple), explanations borrowed from that article:

  • OneHot — one column for each value to compare vs. all other values.
  • Binary — convert each integer to binary digits. Each binary digit gets one column.
  • Hashing — Like OneHot but fewer dimensions, some info loss due to collisions.
  • Backward Difference — the mean of the dependent variable for a level is compared with the mean of the dependent variable for the prior level.
  • Target — use the mean of the dependant variable.

The above-mentioned methods can be used pretty much for all algorithms. And each single has its pros and cons. Some you have more information loss than the other and so on. Good news is that most of these methods are quite easy to use, e.g. via this Python package.

Another method that I found quite interesting and is suitable for Neural Networks is:

  • Entity Embeddings - An special encoding method borrowed from NLP, in which an embedding space is learned on the fly for each categorical variable.

See these blogpost 1, 2 ,3. I am writing an up-and-running code in Google Colab to fully demonstrate this, but you still find pieces of code in those blogposts to give it a try. I find this method way better than for example OneHot for Neural Networks.

Hope these help!

  • $\begingroup$ , Thanks for the detailed answer. Can methods like entity embedding work with the scenario i have mentioned ? it seems that to encode this data : ['fever','pain','yellow skin' .... ] , i need to split each symptom to a single feature and then embed in relation to all the values in that new feature, but the connection between fever , pain and yellow skin will be gone .Remember that ['fever','pain','yellow skin' .... ] is multiple observations for a one patient $\endgroup$
    – Latent
    Jan 15 '19 at 8:17
  • 1
    $\begingroup$ I have not thought like this. When I think now, the entity embedding should be able to capture the connection between symptoms even when they are split. My intuition is that you learn the weights (embedding matrix) of each categorical in association (in a nonlinear way using activation functions in NNs) with other features on the fly as you train your network during backpropagation. The good part is that you can look at embedding spaces in 2D or 3D using a dimensionality reduction to see what happens meaning that if relevant symptoms are closer together, thus it has done what you wanted! Try. $\endgroup$ Jan 15 '19 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.