I understand how a random forest algorithm works but could someone tell me the rationale behind Random Forest feature selection being biased towards high cardinality features?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ there are different ways to calculate feature importance and the answer depends on the implementation. anything in particular you are referring to? $\endgroup$– oW_ ♦May 14, 2019 at 23:02
-
$\begingroup$ I am specifically talking about Random Forest variable importance. $\endgroup$– Jason MathewsMay 14, 2019 at 23:17
-
$\begingroup$ There are different ways to calculate feature importance in random forests - variance and permutation importance are two examples of techniques. $\endgroup$– bradSMay 15, 2019 at 10:17
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Random forest (as almost any other algorithm) is prone to selecting variables which can lead to a one-to-one relationship with the $Y$ variable.
Why? Because you are leading them to be overfitted. If your variables have high cardinality, it means they form little groups (in the leaf nodes) and then your model is "learning" the individuals, not generalizing them. The more "cardinal" the variable, the more overfitted is the model.
For example, if you have social security number as variable (biggest cardinality possible), this variable will for sure have the biggest feature importance. Because in the leaf nodes you will find every individual with his social sec. number and his output. But the capacity of generalization of the model is zero.
Almost every task in data science looks that this doesn't happen.
answered May 15, 2019 at 1:19