My understanding of Random Forest Regression is that each leaf node contains one or multiple examples from the training data. When predicting, each tree finds the most appropriate leaf and takes the mean of the target values of the examples, then the forest takes the mean of the values from each tree. Is that correct?

Now to my specific example: Using RandomForestRegressor from sklearn I am using the following parameters:

    'n_estimators': 100, 
    'max_depth': 9, 
    'max_features': 1 / 3, 
    'criterion': 'mse', 

The training data is in the range between -2.3 and 2.3[1]. After training I predict for new values and I get an output of 33.5. If my understanding of Random Forest models is correct, that should be impossible because it should only be able to predict values in the range of the training data.

To make it even weirder, if I rescale the training data so that it's in the range between 0 and 2.3[2] and train on that, the predictions are all in the expected range.

Can anyone figure out what could be causing this issue?

[1] Strictly speaking the raw data is in the range 0.1 to 10, which is put through an ln transform to get -2.3 to +2.3. Normally I wouldn't use a logarithmic transform on data in this range, but this is a subset of a larger dataset which can have anything from 0.1 to 100,000.

[2] I round all the raw values less than 1 to 1, so after the logarithmic transform it's 0 to +2.3.

  • 1
    $\begingroup$ Have you tried to visualize the decision tree with a graph? It would show you clearly why you have this output with the different potential boundaries. $\endgroup$ Jul 31 at 8:13
  • 1
    $\begingroup$ There is not enough info provided to answer; please post the summary statistics of your data, as well as the specific code used for feeding them into the RF model (the RF parameters are largely irrelevant to the question). $\endgroup$
    – desertnaut
    Jul 31 at 20:14
  • 1
    $\begingroup$ You are correct that this should not happen. To debug why it is, we'd need more details, preferably a reproducible example. $\endgroup$ Aug 2 at 0:52