decision trees on mix of categorical and real value parameters

Question

I have about 3,000,000 samples and each sample is described by a list of size about 20. Some elements in this list are categorical, for example name of cities, day of week, etc. (some categories have a large number of options, for example one category is url with more than 700,000 unique elements in my dataset!). Also some elements have real values, for example for time of day.

My data is labelled (2 categories,) and I need to train a classifier for test data. I am inclined towards decision tree or random forest since they seem to be a good choice for this type of problem.

Now my questions are:

1) How do I pre-process categorical data? one-hot-encoding seem to be the right choice but given that some of my categories have huge number of possible values, one hot encoder will produce very long words! am I correct?

2) How do I combine data from different categories? For example data from category 'cities' with data from 'urls', since they have different lengths. Do I simply concatenate them?

3) How can I combine categorical data with real valued data, for example 'name of cities' with 'time of day' to produce one matrix that can then be passed to decision tree classifier?

4) Are there any special normalisation, etc. that I have to do before passing data to classifier?

I plan to use python and Scikit for this task.

Many thanks for your help.

Answer 1

Unfortunately scikit-learn does not handle categorical features well - they must be encoded. One-hot-encoding is great and can be implemented in sklearn very easily. So if you have an array of feature columns, X, and a class label vector, y

from sklearn import OneHotEncoder
ohe = OneHotEncoder(categorical_features=[0])
X = ohe.fit_transform(X).toarray()

will perform the encoding for you (The categorical_features attribute gives the index of the feature you want encoded). More to your point though, if your feature has many levels, one-hot encoding can leave you with a sparse and inefficient X array. In this case it may be beneficial to do a count transform, possibly replacing each level with its corresponding log-odds ratio.

Count Transforms

The link above does a more thorough job explaining it, but, in layman's terms, for each categorical feature you:

1. count the number of times a level(category) belongs to class0 or class1
2. add 0.5 to each count (this takes care of instances when a level belongs strictly to one class)
3. calculate the probability that it belongs to either class.
4. calculate the log of the odds ratio
5. replace each level with its corresponding log odds ratio

In this manner you get numerical values representing each of the feature's many levels without creating a dummy variable for each level. Still, 20 variables is a lot, and you may want to use PCA for data compression and dimension reduction.

One of the great things about tree based methods in general is that they do not require standardization. That being said, from a data science perspective, I would train several different models (tree based methods, svms, logistic regression, knn, etc.), and see which one yields the best results. All methods besides CART benefit from scaling, so try

from sklearn.preprocessing import StandardScaler
stdsc = StandardScaler()
X_train_std = stdsc.fit_transform(X_train)
X_test_std = stdsc.transform(X_test)

I hope that answers most of what you're asking. I'm not sure why you would want to concatenate two very different variables like cities and urls, but if you encode them or do a count transform your X array should be just fine and you can leave them separated. Also, you shouldn't have to worry about the length of a given variable - python is object oriented (if you're getting error messages complaining about 'length', you've likely just confused your indices).