I'm using sklearn's
RandomForestRegressor to try and model a relationship that involves three Feature variables (
x1,x2,x3) and one Target variable (
y1). My model appears to pass some basic validation-criteria tests, however, when I push fresh data into the model the results are way off from what I'd expect.
Building & Validating the Model
My training data looks like this (see image #1). In this visualization, each X-axis represents a different Feature and the Y-axis represents the same Target data for all three plots. This data is an array of size [658,3] prior to any test-train splitting.
Here are the important bits of the code I'm using to build the model:
X = np.vstack((x1, x2, x3)) X = np.ndarray.transpose(X) # Input columns y = y1 # Target data X_train, X_test, y_train, y_test = train_test_split(X, y, test_ratio) # 0.2 test ratio conversion_model = RandomForestRegressor(n_estimators=150, random_state=0) conversion_model.fit(X_train, y_train)
Running this model through sklearn's
cross_val_score function, with a
cv=10, produces an average cross validation score of ~0.99. Here are some other statistics of the model when I crunch the predicted (train) vs. true (test) results.
- RMSE = 0.005
- Max Error = 0.017
- R2 = 0.999
As far as I can tell, this looks like a pretty happy model.
Predicting Values Using the Model
Here's where the issues arise. I'm feeding the following dataset (see image #2) into this newly generated Random Forest model. In the image, the X-axes are the
x2 features, respectively; the Y-axis is the
x3 feature. Just using Image #2 as a means of sanity checking my experimentally-collected data.
It's also useful to note that Features
x2 do not vary significantly between the model training data and the prediction data. Feature
x3 is the main term that will shift around b/w the training set and the prediction set.
The code I'm using to capture predictions for the new Feature data is:
Xpred = np.vstack((x1_new, x2_new, x3_new)) Xpred = np.ndarray.transpose(Xpred) ypred = conversion_model.predict(Xpred)
When I plot the resulting predictions I end up with something like this (see image #3). Obviously, there is something funky with these results. The lower-end datapoints look okay (?) but the remainder of the prediction data is plateauing and definitely not following the expected trend.
Could really use some help identifying my issue here. Thanks!