I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature importance report shows 5 of these one hot encoded column in top 10, with one of them appearing at the top.
I tried to see if there was a diffrence in percentage of these categories between my binary classes (1, 0) but there isn't so i am a little perplexed as to why it is assigning such a high feature importance to them.
i have read online that if we have a categorical variable with q levels, the tree has to choose from ((2^q/2)-1) splits. For a dummy variable, there is only one possible split and this induces sparsity
i am not sure i understand this, say i have a column called color: red, green, blue,yellow , and i implement one hot encoding so is the number of splits that happen is 2^4/2 -1 = 3? if this increases as i have e.g. 2^25/2 -1, more splits means the tree is more likely to find a 'good split' for data at hand and lead to overfitting? But what i don't understand is how this splitting chages with dummy variables.. does that equation hold or not for one hot endoded variables.
am i interpreting this correctly?
sources elemts of statisicatl learning: