1
$\begingroup$

I am wondering how random forests are exactly implemented in Weka. This paper is very specific about RFs in Weka, but the description of its learning process in chapter 2 seems strange to me. They say:

  1. Bootstrap samples $B_i$ for every tree $t_i$
  2. A random subset of features is selected for each $t_i$
  3. Information gain is used to grow unpruned trees $t_i$

My questions:

  • Shouldn't step 2 be repeated on all levels of the decision tree? Otherwise each tree will never see some of the features
  • Whats the default when setting numFeatures=0. I think this is the number of features that is available for each split. Is it the square root of the number of all features?
  • Is really information gain used for determining the best split attribute?

I am using Weka 3.8.3 - not sure if this matters.

Thanks for all hints :)

$\endgroup$
3
$\begingroup$

Your linked paper appears to be wrong about feature subsetting. I couldn't find it in the documentation for randomForest, but the source for randomForest uses randomTree for the base models, and in that documentation it says

a tree that considers K randomly chosen attributes at each node.

So the selection seems to happen at each split.
(Note that xgboost has feature subsetting at each of the tree, the level (depth), and the node. I don't see any obvious reason that one or more of these options should always be preferable...)

For default number of features, sqrt(m) is the most common, but it looks like Weka uses lg(m). See option -K at
https://weka.sourceforge.io/doc.dev/weka/classifiers/trees/RandomForest.html

Yes, Weka uses the Quinlan family of decision trees, which split using information gain (as opposed to CART, which uses gini).

$\endgroup$
2
  • $\begingroup$ Thanks, this answers the last two questions perfectly. Regarding my first question: Thanks for pointing out all the different ways for feature subsetting. For my work the implementation in Weka is the crucial part. So can you confirm that the paper is correct and one fixed subset of features is used per tree? Maybe you know other references with information about it... $\endgroup$ – Big M Feb 20 '20 at 8:47
  • $\begingroup$ @BigM, It wasn't apparent from the randomForest docs I was looking at. Looking into the source found an answer: the paper you reference appears to be wrong (at least in current Weka). Answer updated. $\endgroup$ – Ben Reiniger Feb 20 '20 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.