2
$\begingroup$

I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is split equally among base classifiers for majority voting. Nemenyi test shows, that majority voting is significantly better (for 11 benchmarking datasets from keel).

From what I understand, the difference between those two methods is that data used to train random forest (base classifiers) might not sum up to the whole dataset. Is my understanding correct? If so, what might be the reason for the observed difference?

Also, could you point me to any articles comparing those two methods?

Edit: If someone was interested in this topic, here's an article comparing bagging with horizontal partitioning in favor of the latter.

$\endgroup$
  • $\begingroup$ What are the common depth, number of trees, ...? What is the shape of your data? $\endgroup$ – Ben Reiniger Oct 18 at 0:48
1
$\begingroup$

Random forests' base-learner trees use "bootstrapping," by default with rate 1.0 (parameter subsamplingRate); that is, the dataset is resampled but with replacement. So each tree learns on a dataset of the same size as the original, but with some of those points duplicated and some left out. For large datasets, it works out to be about 1/3 of the datasets are left out for each tree. With enough trees (really, just a few is enough), it becomes extremely unlikely that any datapoint is never used by any of the trees.

Spark appears to use hard voting for its random forests, so that's not the difference.

It seems to me that the main difference here is that you've partitioned the data for your custom implementation, so those base learners learn on substantially less data. If that's doing well, it suggests that the random forest is overfitting in comparison. I would suggest varying the tree parameters, say by making the trees in the random forest more conservative, to see how they compare then.

$\endgroup$
  • $\begingroup$ Thank you, have you stumbled on any paper that compares those two methods? I'm having difficulty finding one. $\endgroup$ – Andronicus Oct 18 at 4:13
  • $\begingroup$ I hadn't heard of partitioning the data for ensembling before; but a quick search found: www3.nd.edu/~nchawla/papers/ICDM01.pdf . $\endgroup$ – Ben Reiniger Oct 18 at 14:13
  • $\begingroup$ sorry, but I cannot see decision tree as a reference there $\endgroup$ – Andronicus Oct 20 at 8:32
  • $\begingroup$ ? The phrase "decision tree" is used at least 5 times, and several other times the specific learning algorithm "C4.5" is mentioned. $\endgroup$ – Ben Reiniger Oct 20 at 19:14
1
$\begingroup$

Random forest, predicts the class with highest probability estimate. The predicted class probabilities of an input sample is computed as the mean predicted class probabilities of the trees in the forest. The class probability of a single tree is the fraction of samples of the same class in a leaf.

Majority voting, which is also called Hard Voting, every individual classifier votes for a class, and the majority wins. In statistical terms, the predicted target label of the ensemble is the mode of the distribution of individually predicted labels.

Majority voting may works better in cases where there are some outliers. Consider these votes: $\{0.51, 0.51, 0.51, 0.01\}$ and $\{1,1,1,0\}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.