# Decision Tree - Preprocessing for very sparse features

How do we pre-process data for very sparse features for a decision tree?

It mentioned this:

Why chose decision trees? Different kinds of models have different advantages. The decision tree model is very good at handling tabular data with numerical features, or categorical features with fewer than hundreds of categories. Unlike linear models, decision trees are able to capture non-linear interaction between the features and the target.

One important note is that tree based models are not designed to work with very sparse features. When dealing with sparse input data (e.g. categorical features with large dimension), we can either pre-process the sparse features to generate numerical statistics, or switch to a linear model, which is better suited for such scenarios.

If we have a categorical column that is very sparse (many 0's), then for that category what type of numerical statistics can we transform for each row?

• When he says sparse features he is talking about high cardinality, right? First of all, for sparse I understand that he is talking about OneHotEncoder, and you dont necessarily must preprocess data in this way to work with categorical features in tree models. The problema with high cardinality is overfitting, as you have so much categorical labels, the split points on trees will be affected and the distribution from train/test split should be different. Talking about statistics, one way you could work calculating target mean encodings. I am lcommenting because I dont know if it answers your qu Commented Mar 7, 2019 at 14:34
• @VictorOliveira I mean I'm not sure what he is saying regarding statistics. If we did generate mean encodings, is this ordinal that we can put it in as a continuous variable instead of one hot encoding? If you think creating mean encoding is what he means then please put it as an answer because I never thought about this approach! Commented Mar 8, 2019 at 5:29
• Maybe by "numerical statistics" he means using numerical methods like SVD/PCA to do dimensionality reduction to reduce the number of sparse features. Commented Sep 28, 2020 at 9:48
• What the author meant is that decision tree output fits better for sparse categories. So if there are a lot of 0s, it will try and not miss the sparsely encountered 1s. Contrast that with neural networks which won't really care about the 1s since it already got a good amount of accuracy by marking everything as 0s. Commented Mar 1, 2021 at 11:10