47
$\begingroup$

I am new to machine learning!

Right now I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-learn takes only numbers as parameters, but I want to inject the strings as well as they carry a significant amount of knowledge.

How do I handle such a scenario?

I can convert a string to numbers by some mechanism such as hashing in python. But I would like to know the best practice on how strings are handled in decision tree problems.

Thanks for your support!

$\endgroup$
  • $\begingroup$ In case of sckitlearn I have seen that we need to encode the categorical variables, else fit method would throw an error saying ValueError: could not convert string to float $\endgroup$ – Kar Aug 31 '16 at 23:53
45
$\begingroup$

In most of the well-established machine learning systems, categorical variables are handled naturally. For example in R you would use factors, in WEKA you would use nominal variables. This is not the case in scikit-learn. The decision trees implemented in scikit-learn uses only numerical features and these features are interpreted always as continuous numeric variables.

Thus, simply replacing the strings with a hash code should be avoided, because being considered as a continuous numerical feature any coding you will use will induce an order which simply does not exist in your data.

One example is to code ['red','green','blue'] with [1,2,3], would produce weird things like 'red' is lower than 'blue', and if you average a 'red' and a 'blue' you will get a 'green'. Another more subtle example might happen when you code ['low', 'medium', 'high'] with [1,2,3]. In the latter case it might happen to have an ordering which makes sense, however, some subtle inconsistencies might happen when 'medium' in not in the middle of 'low' and 'high'.

Finally, the answer to your question lies in coding the categorical feature into multiple binary features. For example, you might code ['red','green','blue'] with 3 columns, one for each category, having 1 when the category match and 0 otherwise. This is called one-hot-encoding, binary encoding, one-of-k-encoding or whatever. You can check documentation here for encoding categorical features and feature extraction - hashing and dicts. Obviously one-hot-encoding will expand your space requirements and sometimes it hurts the performance as well.

$\endgroup$
  • 2
    $\begingroup$ It's the scikit implementation that it doesn't handle categorical variables properly. Recoding like how this answer suggests is probably the best you can do. More serious user might look for alternative package. $\endgroup$ – SmallChess Sep 18 '16 at 1:46
  • 3
    $\begingroup$ One can use sklearn.preprocessing.LabelBinarizer for one-hot-encoding of categorical variable. $\endgroup$ – crackjack Nov 9 '16 at 9:29
  • $\begingroup$ @rapaio I think binary coding is not same one hot encoding. Binary coding is when you represent 8 categories with 3 columns, or between 9 to 16 categories with 4 columns and so on. Am I wrong? $\endgroup$ – Alok Nayak Apr 21 '17 at 10:59
  • $\begingroup$ the patsy python package will deal with one-hot encoding of categorical variables. patsy.readthedocs.io/en/latest/quickstart.html $\endgroup$ – zhespelt May 17 '17 at 18:13
  • 3
    $\begingroup$ Do not use LabelBinarizer, use sklearn.preprocessing.OneHotEncoder. If you are using pandas to import and pre-process your data, you can also do that directly using pandas.get_dummies. It sucks that scikit-learn does not support categorical variables. $\endgroup$ – Ricardo Cruz Jul 28 '17 at 9:52
11
$\begingroup$

You need to encode your strings as numeric features that sci-kit can use for the ML algorithms. This functionality is handled in the preprocessing module (e.g., see sklearn.preprocessing.LabelEncoder for an example).

$\endgroup$
  • 3
    $\begingroup$ rapaio explains in his answer why this would get an incorrect result $\endgroup$ – Keith Apr 25 '17 at 16:14
4
$\begingroup$

You should usually one-hot encode categorical variables for scikit-learn models, including random forest. Random forest will often work ok without one-hot encoding but usually performs better if you do one-hot encode. One-hot encoding and "dummying" variables mean the same thing in this context. Scikit-learn has sklearn.preprocessing.OneHotEncoder and Pandas has pandas.get_dummies to accomplish this.

However, there are alternatives. The article "Beyond One-Hot" at KDnuggets does a great job of explaining why you need to encode categorical variables and alternatives to one-hot encoding.

There are alternative implementations of random forest that do not require one-hot encoding such as R or H2O. The implementation in R is computationally expensive and will not work if your features have many categories. H2O will work with large numbers of categories. Continuum has made H2O available in Anaconda Python.

There is an ongoing effort to make scikit-learn handle categorical features directly.

This article has an explanation of the algorithm used in H2O. It references the academic paper A Streaming Parallel Decision Tree Algorithm and a longer version of the same paper.

$\endgroup$
2
$\begingroup$

Turn them to numbers, for example for each unique country assingn a unique number (like 1,2,3 and ...)

also you Don't need to use One-Hot Encoding (aka dummy variables) when working with random forest, because trees don't work like other algorithm (such as linear/logistic regression) and they don't work by distant (they work with finding good split for your features) so NO NEED for One-Hot Encoding

$\endgroup$
2
$\begingroup$

2018 Update!

You can create an embedding (dense vector) space for your categorical variables. Many of you are familiar with word2vec and fastext, which embed words in a meaningful dense vector space. Same idea here-- your categorical variables will map to a vector with some meaning.

From the Guo/Berkhahn paper:

Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features.

The authors found that representing categorical variables this way improved the effectiveness of all machine learning algorithms tested, including random forest.

The best example might be Pinterest's application of the technique to group related Pins:

enter image description here

The folks at fastai have implemented categorical embeddings and created a very nice blog post with companion demo notebook.

$\endgroup$
1
$\begingroup$

You can use dummy variables in such scenarios. With panda's panda.get_dummies you can create dummy variables for strings you want to put in Decision Tree or Random Forest.

Example:

import pandas as pd
d = {'one' : pd.Series([1., 2., 3.,4.], index=['a', 'b', 'c','d']),'two' :pd.Series(['Paul', 'John', 'Micheal','George'], index=['a', 'b', 'c', 'd'])}
df = pd.DataFrame(d)

df_with_dummies= pd.get_dummies(df,columns=["two"],drop_first=False)
df_with_dummies
$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.