# What is the implication of having features with less variation in a tree based model?

I'm training a tree-based model (e.g. xgb). I have some features with more than 90% values constant. Does it add value to the model since the variation in the data is minimal?. What would be the impact of the same if I were to use a linear regression model?

Variation is not the key. Notice that 0/1 indicator variables are used frequently and might have mostly 0's (like many missing indicators). The key is where is the variation in relation to what you are predicting and in relation to interactions.

For example, if your column is 0 where target = 0 and not 0 where target = 1, then the variation does not matter. Add a new indicator feature where the original column is 0 or not 0 may be a good way. You might want to do this anyway.

Also with trees, columns are interacting, so if the lack of variation in the column highlights better predictive power in another column, then this is a win. Again, a transformation to an indicator variable may be useful.

Similar to a linear model. You add the interactions yourself with linear models.

Of course, the column may not be useful at all to the model, lack of variation or not.

No way to know unless you try with your data.

• Thanks, @Craig. "transformation to an indicator variable may be useful". Can you elaborate on this? Jul 28 at 12:17
• "You add the interactions yourself with linear models". In Linear models, also features interact, right? The presence of one feature can affect other features. Why do we have to add interactions ourselves Jul 28 at 12:20
• Let's say your current column with 90% the same values = 0 for 90% and != 0 for 10%. Make a new column that has 0 for the 0 rows and a 1 for the non-0 rows. Or a 0, -1, +1 for original 0,<0,>0. Perhaps the strongest signal that the model can find is the constant value or not the constant value - feature engineering!! To learn the basics about interactions in linear models here are 2 articles - cantab.net/users/filimon/cursoFCDEF/will/logistic_interact.pdf and yury-zablotski.netlify.app/post/…. Hope this helps. Jul 28 at 13:03

Variation does not matter what matters is how good is your variable in predicting the target variable.

in order to know whether linear model is applicable, you have to fit data on linear classification . If the accuracy is good then you can proceed with the linear classification else consider more complicated models like SVM decision tree