How to create classification decision trees on a dataset that has both numerical and categorical variables?

Question

I am quite new to Data Science and learning things hands-on in the job. I am a fraud analyst and my job is to predict whether an application is fraudulent or not based on data.

Before moving on to many advanced models, I am asked to build decision trees on the dataset. Now the dataset which I have has 1500 columns; some categoricals and some numeric. Different categorical variables have different levels; some are binary and some have 100+ levels.

I came across the fact that scikit-learn can work only if the entire dataset comprises numeric variables (discrete or continuous). And the frequent work-around that I am seeing is around one-hot encoding like here - which I do not believe is pragmatic in my case because of a sheer number of columns and levels.

I asked my bosses to give me few weeks to understand most of the data so as to limit my variables and possibly do one-hot encoding but that's not flying well with them.

Has anyone any experience building classification decision trees on a mixed datatype dataset with large counts of variables?

Thanks.

Answer 1

1.) Before encoding your categorical features, you should probably do some feature engineering, so as to reduce the number of categorical features. Once you are sure that you cannot reduce them further without losing valuable info, only then encode them using one hot encoder or any other technique.

2.) You can use the drop feature of One Hot Encoder to reduce the encoded columns. For example the feature gender has 2 dimensions male and female. When you encode them you'll get gender_male and gender_female. If you use the drop feature when encoding, either one of the dimensions will be dropped as the model can work with 1 dimension for the gender feature. This will further reduce the dimensionality of your model.

3.) Other thing you can do is for every feature you can select the top 3-4 most frequent values, drop the others and then encode them. As you mentioned you have some features that have 100+ values. You can choose top 10-20 values for that feature, drop the others and then one hot encode. This will reduce your dimensions by 80% for that variable.

4.) You can check out PCA which is used for dimensionality reduction purposes. I do not have enough knowledge on it.

Other than you have to live with the fact that your model will contain high dimensionality (curse of dimensionality!). This haunts each and every Data Scientist.

Answer 2

What you mentioned is true, for 99% of Scikit-learn's estimators, the data X must be numeric (I think only HistGradientBoosting works with no numerical categorical data) So when working with mixed data types in modeling, Pipelines + ColumnTransformers the answer always is.

Try something like this and it would worked no matter the type you have:

# You may want to change the preprocessing steps for both numerical and categorical

from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import Pipeline
from sklearn.compose import make_column_transformer, make_column_selector as selector
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer


cont_prepro = Pipeline([("imputer",SimpleImputer(strategy = "median")),("scaler",StandarScaler())])

cat_prepro = Pipeline([("imputer",SimpleImputer(strategy = "most_frequent")),("encoder",OneHotEncoder(handle_unknown = "ignore"))])

preprocessing = make_column_transformer((cont_prepro,selector(dtype_exclude = "object")),(cat_prepro,selector(dtype_include = "object"))

pipe = Pipeline([("preprocessing",preprocessing),("model", DecisionTreeClassifier())])

pipe.fit(X_train, y_train)

If dimensionality of your input matrix is a concern, you can even include a feature selector inside the pipeline like:

# adjust the parameters of kbest
from feature_selection import SelectKBest

pipe = Pipeline([("preprocessing",preprocessing),("selector",SelectKBest()),
("model", DecisionTreeClassifier())])

Just be careful if you want to use PCA, you should only use this in the numerical features, not in the categorical ones