0
$\begingroup$

I will try to draw a little context to my question from title.

I build a Random Forest Regressor from 1000 trees using sklearn. Then I exported all the decision paths along with predicted values for each Path and tree (around 50k decision paths, as one tree includes many decision paths).

I want to predict value for a particular sample. I can't use sklearn method predict() for some reasons but I have its value for comparison. I expected that I will take all decision paths for which the data from my sample meet conditions and calculate average to get final predicted value.

For example, if 3 decision paths corresponds to values from my sample, I could calculate average from 3 predicted values to get the final one.

But result from this operation differs a lot from value obtained by predict() method.

I would be grateful if someone could explain how this regressor works. And the second question is, is there any other way to obtain predicted value based only on decision paths of trees in the regressor.

Thank you very much!

And sorry for not attaching any code. I can clarify the description more if needed :)

$\endgroup$

1 Answer 1

0
$\begingroup$

The Random Forest Regressor works by creating a multitude of decision trees at training time and outputting the mean prediction of the individual trees at prediction time.

When you use the predict() method, it's not just taking the average of the predictions of the decision paths that your sample meets the conditions for. It's taking the average of the predictions of all the trees in the forest. Each tree in the forest is making a prediction for your sample, regardless of whether the sample meets the conditions for a particular decision path in that tree.

So, if you want to replicate the predict() method using only the decision paths, you would need to do the following:

  1. For each tree in the forest, find the decision path that your sample meets the conditions for.
  2. Get the prediction for that decision path.
  3. Repeat this for all trees in the forest.
  4. Take the average of all these predictions.

This should give you the same result as the predict() method. If it doesn't, there might be an issue with how you're determining which decision paths your sample meets the conditions for, or with how you're calculating the predictions for those decision paths.

Also, each tree in the forest is built on a different subset of the training data, and each split in each tree is based on a different subset of the features. So, it's possible for a sample to meet the conditions for a decision path in one tree but not in another, even if the decision paths look very similar.

$\endgroup$
5
  • $\begingroup$ Thank you for your clear answer. As I understood, every tree should be able to return predicted value for my sample, even if conditions are not met. My question to that - what value will be returned in that case? I mean, if data from sample doesnt follow decision paths in some tree, what predicted value that situation leads to? $\endgroup$
    – Paulina
    Jun 29, 2023 at 19:20
  • $\begingroup$ Every decision tree in a random forest is a complete model, meaning it can provide a prediction for any given sample. The tree does this by traversing from the root to a leaf node based on the feature values of the sample. Each internal node in the tree represents a decision point (a condition on a feature), and the tree uses these conditions to decide which path to follow for a given sample. $\endgroup$ Jun 29, 2023 at 19:47
  • $\begingroup$ So, even if your sample's feature values don't exactly match the conditions at every decision point, the tree will still make a decision based on the conditions that are defined. It will follow the path that corresponds to the best match for the sample's feature values. Once the tree reaches a leaf node, it returns the average target value of the training samples that ended up in that leaf node. This is the predicted value for the given sample. $\endgroup$ Jun 29, 2023 at 19:47
  • $\begingroup$ Thank you. I needed to check my code once again and everything that you wrote, had found confirmation in my fixed code. After creating the right decision paths, every tree become a separate estimator and was able to return the predicted value. $\endgroup$
    – Paulina
    Jul 3, 2023 at 13:45
  • $\begingroup$ Happy to help you $\endgroup$ Jul 3, 2023 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.