1
$\begingroup$

Everything that I've read about random forests has indicated that they do not require scaling of inputs and that scaling should not affect the construction of the model. Here's a quote from another SE question (https://stats.stackexchange.com/questions/255765/does-random-forest-need-input-variables-to-be-scaled-or-centered):

Random Forests are based on tree partitioning algorithms.

As such, there's no analogue to a coefficient one obtain in general regression strategies, which would depend on the units of the independent variables. Instead, one obtain a collection of partition rules, basically a decision given a threshold, and this shouldn't change with scaling. In other words, the trees only see ranks in the features.

Basically, any monotonic transformation of your data shouldn't change the forest at all (in the most common implementations).

Here's what I'm using currently. If I remove the weights multiplier, I get a different model (i.e., different value from model.score and different tree depths) despite setting random_state=0 in either case.

model = RandomForestRegressor(n_estimators=10, criterion='mse', random_state=0)
weights = np.arange(1,self.x_train.shape[1]+1)[None,:]
# weights = [[ 1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15.]]
model.fit(self.x_train * weights, self.y_train)

By comparison, I noticed that if I use XGBRegressor rather than RandomForestRegressor, scaling does not change the model.

Is there an obvious mistake that I'm making or is the explanation above not correct?

$\endgroup$
  • $\begingroup$ What exactly do you mean by "I get a different model"? Different score on a test set, different model attributes (which?), ...? $\endgroup$ – Ben Reiniger May 15 at 21:52
  • $\begingroup$ I'm running a custom test (and getting different results), but I just checked the results using model.score and get_depths on the individual trees and they are both different. $\endgroup$ – SuperCodeBrah May 15 at 22:05
  • $\begingroup$ I agree that a linear transformation of each predictor shouldn't cause a change in the model, but I don't see an obvious mistake. Could you provide a MWE? (Is self.x_train consistent?) $\endgroup$ – Ben Reiniger May 16 at 2:05
  • 1
    $\begingroup$ I figured it out what's causing the issue but not why (see my answer). Any ideas? $\endgroup$ – SuperCodeBrah May 16 at 5:17
1
$\begingroup$

After trying to recreate the issue with random numbers (and failing initially), I figured out that the problem comes from the fact that the x_train data that I'm using contains columns that have a very small, near-zero values.

To recreate, the first section is only run once:

scale = 0.0001 # making this larger eliminates the issue
x_train = np.random.uniform(0,scale,size=(1000,10))
y_train = np.random.uniform(0,1,size=(x_train.shape[0]))

Then using the same values for x_train and y_train, run the section below but with use_weights set to True and then False.

use_weights = True
model = RandomForestRegressor(n_estimators=10, random_state=0)

if use_weights:
    weights = np.arange(1,x_train.shape[1]+1)[None,:]
    model.fit(x_train * weights, y_train)
    prediction = model.predict(x_train * weights)
else:
    model.fit(x_train, y_train)
    prediction = model.predict(x_train)

print(prediction[0]) # changes based on use_weights value assuming scale is very small

As a side note, the y_train values for the real dataset are also very small and I have to multiply them by 100 or more to get the model to even run. That is, it will not create any leaves at all without expanding the scale of the y_train values (confirmed by running the get_depth method on each tree).

I'm wondering, is this purely a numerical imprecision issue or is it something unique to the random forest calculations that are happening under the hood?

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.