I have used both weka random forest and sklearn random forest in my research, but I have realised that they use different methods to combine the predictions of the base learners i.e. decision trees to make the final prediction. To predict the class of an instance, weka random forest uses majority vote which predicts the class of the instance as the class predicted by majority of the decision trees. The class probability of the instance is computed as fraction of the no. of the trees that predict that class to the total no. of the trees of the random forest.
The sklearn random forest predicts the class of an instance as follows. The predicted class of an input instance is the class with the highest mean probability estimate across the trees. The predicted class probabilities of an input instance are computed as the mean predicted class probabilities of the trees in the forest. The class probability of a single tree is the fraction of samples of the same class in a leaf. https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba
I think the class probability of weka random forest (majority vote) is correct but the class probability of sklearn random forest does not seem correct. Because for sklearn random forest, the sizes of classes of the training set determines the class probabilities of a single tree and the class probabilities of the random forest. So, removing instances from or adding instances to the training set would change the class probabilities of the random forest which does not seem correct. The majority vote is not affected by the sizes of the classes of the training set.
However, the cross validation performances of weka random forest and sklearn random forest are similar. For example, the 10-fold cross validation results of these 2 random forest approaches on 5 datasets from UCI database are as follows:
|dataset||10-fold CV AUC of weka random forest||10-fold CV AUC of sklearn random forest|
I did another experiment for the diabetes and sonar datasets using weka random forest and sklearn random forest respectively:
- split the dataset into a training set (80%) and a test set (20%) using stratified sampling.
- Oversample the training set into a balanced data set of size 1000 using random sampling with replacement.
- Train a random forest with 50 trees
- Test the random forest on the test set
- Repeat steps 1 to 4 for 100 times
The results of weka random forest and sklearn random forest are very similar in terms of average testing AUC over 100 times.
|dataset||average test AUC of weka random forest||average test AUC of sklearn random forest|
Q: Why the performances of weka random forest and sklearn random forest are similar but they use different methods to compute class probabilities of an input instance?
Another main difference is that sklearn random forest cannot be applied to discrete features. The discrete features must be transformed to numeric features by one-hot encoding or ordinal encoding before applying sklearn random forest. Weka random forest can be applied to both numeric features and discrete features directly.