0
$\begingroup$

I'm training a tree-based model (e.g. xgb). I have some features with more than 90% values constant. Does it add value to the model since the variation in the data is minimal?. What would be the impact of the same if I were to use a linear regression model?

$\endgroup$

2 Answers 2

1
$\begingroup$

Variation is not the key. Notice that 0/1 indicator variables are used frequently and might have mostly 0's (like many missing indicators). The key is where is the variation in relation to what you are predicting and in relation to interactions.

For example, if your column is 0 where target = 0 and not 0 where target = 1, then the variation does not matter. Add a new indicator feature where the original column is 0 or not 0 may be a good way. You might want to do this anyway.

Also with trees, columns are interacting, so if the lack of variation in the column highlights better predictive power in another column, then this is a win. Again, a transformation to an indicator variable may be useful.

Similar to a linear model. You add the interactions yourself with linear models.

Of course, the column may not be useful at all to the model, lack of variation or not.

No way to know unless you try with your data.

$\endgroup$
3
  • $\begingroup$ Thanks, @Craig. "transformation to an indicator variable may be useful". Can you elaborate on this? $\endgroup$
    – NAS_2339
    Jul 28 at 12:17
  • $\begingroup$ "You add the interactions yourself with linear models". In Linear models, also features interact, right? The presence of one feature can affect other features. Why do we have to add interactions ourselves $\endgroup$
    – NAS_2339
    Jul 28 at 12:20
  • $\begingroup$ Let's say your current column with 90% the same values = 0 for 90% and != 0 for 10%. Make a new column that has 0 for the 0 rows and a 1 for the non-0 rows. Or a 0, -1, +1 for original 0,<0,>0. Perhaps the strongest signal that the model can find is the constant value or not the constant value - feature engineering!! To learn the basics about interactions in linear models here are 2 articles - cantab.net/users/filimon/cursoFCDEF/will/logistic_interact.pdf and yury-zablotski.netlify.app/post/…. Hope this helps. $\endgroup$
    – Craig
    Jul 28 at 13:03
0
$\begingroup$

Variation does not matter what matters is how good is your variable in predicting the target variable.

in order to know whether linear model is applicable, you have to fit data on linear classification . If the accuracy is good then you can proceed with the linear classification else consider more complicated models like SVM decision tree

New contributor
amol goel is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.