I'm trying to improve the accuracy of a classifier, a random forest one. I built different models with the same hyperparameters but with different random seeds, trained them with the same training data, used the same test daata to make the predictions and compared the results. I discovered that 50% of the errors were always made on the same samples. Therefore, do these samples which are always wrongly predicted deserve a particular attention or is it kind of logic ?

I hope the question is clear enough.

  • $\begingroup$ What do you mean by "on the same sample". It seems that here you have one test set and one training (same sample) set and that you use the seed to randomize the way you train your random forest, am I right ? $\endgroup$ Jul 31 '19 at 17:32
  • $\begingroup$ What I mean is that the ids of the samples which are wrongly predicted are always the same (for 50% of them). Indeed, I have a one test set and one training set. I use the seed for the random_state of the classifier (scikit-learn) $\endgroup$
    – qwertzuiop
    Jul 31 '19 at 17:46

What you are experiencing is pretty normal given your approach. A random forest is an ensemble of decision trees where models are trained in parallel using bootstrapped samples (a technique called bagging). Even though the decision trees are randomized (a thorough explanation of random_state can be found in this question), they still rely on an internal criterion (such as Gini index by default in RandomForestClassifier) to split the nodes and define decision paths. The fact that some of your samples are consistently being misclassified regardless of the random state is an indication of their objective difficulty when using this specific criterion.

Therefore, do these samples which are always wrongly predicted deserve a particular attention or is it kind of logic ?

You are absolutely correct with your first thought. Paying particular attention to wrongly predicted samples in ensembles is the goal of a technique called boosting. The main idea is to train ensemble models in sequence, with new learners focusing on data points that the rest of the ensemble has previously failed on. A great overview of ensemble approaches is presented in this answer, which I highly recommend.

As far as boosting algorithms go, there are different flavors as well: you might want to try sklearn's AdaBoost and gradient tree boosting implementations, or XGBoost. These may help you finally defeat those pesky hard-to-classify samples, but be aware that bagging (your current model) has its own perks that boosting lacks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.