I'm using sklearn's RandomForestRegressor to try and model a relationship that involves three Feature variables (x1,x2,x3) and one Target variable (y1). My model appears to pass some basic validation-criteria tests, however, when I push fresh data into the model the results are way off from what I'd expect.

Building & Validating the Model

My training data looks like this (see image #1). In this visualization, each X-axis represents a different Feature and the Y-axis represents the same Target data for all three plots. This data is an array of size [658,3] prior to any test-train splitting.

Here are the important bits of the code I'm using to build the model:

X = np.vstack((x1, x2, x3))
X = np.ndarray.transpose(X)  # Input columns
y = y1  # Target data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_ratio)  # 0.2 test ratio

conversion_model = RandomForestRegressor(n_estimators=150, random_state=0)
conversion_model.fit(X_train, y_train)

Running this model through sklearn's cross_val_score function, with a cv=10, produces an average cross validation score of ~0.99. Here are some other statistics of the model when I crunch the predicted (train) vs. true (test) results.

  • RMSE = 0.005
  • Max Error = 0.017
  • R2 = 0.999

As far as I can tell, this looks like a pretty happy model.

Predicting Values Using the Model

Here's where the issues arise. I'm feeding the following dataset (see image #2) into this newly generated Random Forest model. In the image, the X-axes are the x1 and x2 features, respectively; the Y-axis is the x3 feature. Just using Image #2 as a means of sanity checking my experimentally-collected data.

It's also useful to note that Features x1 and x2 do not vary significantly between the model training data and the prediction data. Feature x3 is the main term that will shift around b/w the training set and the prediction set.

The code I'm using to capture predictions for the new Feature data is:

Xpred = np.vstack((x1_new, x2_new, x3_new))
Xpred = np.ndarray.transpose(Xpred)
ypred = conversion_model.predict(Xpred)

When I plot the resulting predictions I end up with something like this (see image #3). Obviously, there is something funky with these results. The lower-end datapoints look okay (?) but the remainder of the prediction data is plateauing and definitely not following the expected trend.

Could really use some help identifying my issue here. Thanks!

Image 1: MODEL INPUT DATA VISUALIZATION Visual of the model input data before test-train split. X-axes are the three features and Y-axis is the Target data for all three plots.

Image 2: NEW INPUT DATA, TO BE PREDICTED Visualization of the predictive input data. X-axes are the x1 and x2 features and the Y-axis is the x3 feature.

Image 3: MODEL PREDICTIONS (BAD PREDICTIONS) Predictions using the Random Forest model and the input data in Image 2.

  • $\begingroup$ I gather that Random Forest (and other ensemble methods) will never make predictions that are outside of the range of the original training dataset. Given that the upper bound of my prediction dataset (x3, specifically) is greater than the upper bound of my training dataset; could that explain the plateau response that I'm seeing? $\endgroup$ Jun 23 at 19:30


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.