Random Forest with less samples & variation in test_scores

Question

I'm building a RandomForestRegressor with 75 samples. The distribution of y (After train_test_split) is as below. (Blue-Train and Red-Test)

Keeping test_size=0.3 (hold out) and doing a GridSearchCV on the training set, and initializing a new model using the resulting best_params_, I get a test score of 0.83 on the hold out set.

But when i run this a second time (another random test, train split), the accuracy goes down to even as low as 0.35. I repeated the score check (R^2) 100 times for this model, below is its distribution.

sco =[]
for i in range(0,100):
    X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.30)
    score = best_grid.score(X_test,y_test)
    sco.append(score)
sns.histplot(data=sco)

Assuming that the outliers caused the R2 to vary like this in test set, i decided to remove the outliers (dataset drops to 66 samples) and retrain the model. Below is the distribution of target y after outliers removed.

However following the same steps as above for model, the score drops even further, strangely to 0.20. Iterative run of score check shown below. For most of the tests, the R2 stays below 0.5.

Any Idea why the decline ?

Answer 1

It could be that these "outliers" which you removed were not really the problem, maybe there were in fact easier to predict then the more common cases and increased $R^2$ significantly, and that is why your performance has declined after removing them. Or there could be any number of other reasons.

Answer 2

You have 75 samples which is not enough data for meaningful machine learning. The result is high variance in performance between different runs.