2
$\begingroup$

So I am a beginner in machine learning and just started learning about random trees in this article here. When it talks about tuning the hyperparameter K, I'm a bit confused as to how it works. It says:

The parameter K denotes the number of random splits screened at each node to develop Extra-Trees. It may be chosen in the interval [1, ... , n], where n is the number of attributes.

So K would be the number that determines how many attributes to consider for a random split. Then to split, a random attribute from that set will be chosen? But what I'm wondering is that:

If K > 1, in a given set of attributes [1,2,3,4,...,n], is it always a contiguous subset of size K? Or is it K random attributes chosen from those n attributes? And once you choose a random attribute from that subset, it is replaced or left out?

It also says:

For a given problem, the smaller K is, the stronger the randomization of the trees

I'm confused as to why this is.

$\endgroup$
1
$\begingroup$

So K would be the number that determines how many attributes to consider for a random split. Then to split, a random attribute from that set will be chosen?

No, a random split is made for each of the randomly selected features, and these $K$ potential splits are scored and the highest scoring one is made.

If K > 1, in a given set of attributes [1,2,3,4,...,n], is it always a contiguous subset of size K? Or is it K random attributes chosen from those n attributes? And once you choose a random attribute from that subset, it is replaced or left out?

We choose $K$ random attributes, not necessarily (and probably not) contiguous. I think the original algorithm selects without replacement, and the implementation in sklearn definitely does it without replacement. I could imagine another implementation deciding to do it with replacement though: still the split point for that feature could be different between the two selections. (One more caveat is when a selected feature is constant for the population in the current node: these should be left out, and probably it is implementation-specific whether to select a new feature or just have fewer than $K$ in the end.)

It also says:

For a given problem, the smaller K is, the stronger the randomization of the trees

I'm confused as to why this is.

The smaller $K$ is, the fewer features you consider for a split, and so the more the algorithm depends on these random choices (and is less likely to latch on to noisy, spurious correlations). Compare to a full decision tree model, in which every split point of every feature is considered.

$\endgroup$
0
$\begingroup$

My 2 cents: I strongly recommend displaying ALL PARAMETERS such as criterion, max_depth, max_leaf_nodes... in order to learn tree algorithms. In my experience, you'll learn a ton such as overfitting.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.