I'm learning about decision trees, and I feel like up till now I've understood them and the math behind them pretty well except for one thing: the gain ratio.
As I understand, the gain ratio is implemented in order to punish for features that may take on A LOT of possible values.
If a feature takes on a lot of possible values, it becomes plausible that if we split on that feature there may be values that only point to a single class, but simply because there are only 1 or 2 data points with that value for that feature anyways.
In other words, the only reason that we would get low entropy for splitting on that feature is because the feature could take on a lot of values, and therefore a lot of those values pointed specifically to a single label. So our decision tree algorithm would end up splitting up on something like "ID #", and wrongly calculate that we just had a HUGE information gain.
However, this only seems like a problem because "ID#" is a feature we shouldn't be splitting on to begin with. I mean, if we had another feature that also took on a lot of possible values, but each of those values actually DID imply some label for that datapoint, then wouldn't applying the gain ratio mean that we are actually messing up our decision tree by punishing what was actually a very good split with tons of information gain?
Isn't it better to just identify which feature will have nothing to do with our labeling BEFORE we feed in the training data to the algorithm?
IDK, I just don't see why the gain ratio would really be useful...