Random Forest Overfitting, issues with mtry=1?

Question

I am constructing what is known as an 'Expected Goals' model for football. This metric measures shot quality and a probability is assigned to a shot to achieve this, i.e. the chance a shot will be converted. To create this model I am using a random forest classifier. For evaluation purposes I am only interested in the accuracy of the probabilities rather than strictly classifying shots, therefore, I use the predictions to calculate the Mean Square Error where goal = 1 and no goal = 0. The MSE for the test set along with two benchmarks are as follows:

[1] "test.rf_mse: 0.0856633533734135"
[1] "comparison_model_mse: 0.0820007160001345"
[1] "naive_baseline_mse: 0.0912291249164997"

Note that the comparison model should be better than mine but the naive baseline should be worse. When looking at this the model looks to be doing okay. However, when I apply the same steps to the training set I get the following:

[1] "test.rf_mse: 0.0112001023587005"
[1] "comparison_model_mse: 0.0722459417565357"
[1] "naive_baseline_mse: 0.0858344459279039"

Here the MSE falls to unrealistic levels. Doesn't this mean that my model is overfitting? I understand that the idea that random forests can't overfit isn't strictly correct as all models can overfit to some extent, but for the model to be overfitting by this much must mean I am misunderstanding something here.

#FINAL MODEL FOR SHOTS DATASET
set.seed(5555)
trainIndex <- createDataPartition(shots$goal.miss, p = .75, list = FALSE)
train_set <- shots[ trainIndex,]
test_set  <- shots[-trainIndex,]

set.seed(1000)
rf.shots <- randomForest(as.factor(goal.miss) ~ ., data=train_set, ntree=500, mtry=5)

###TEST RESULTS
pred <- predict(rf.shots, test_set, type="prob")
test_set$predictions.test <- pred[,2]
mean(((test_set$goal.miss - test_set$predictions.test)^2))

###TRAIN RESULTS 
pred <- predict(rf.shots, train_set, type="prob")
train_set$predictions.train <- pred[,2]
mean(((train_set$goal.miss - train_set$predictions.train)^2))

NOTE: I moved on to using the 'ranger' random forest package which allowed me to tune more hyperparameters. From a grid search, I found the only parameter that appeared to cause overfitting was mtry. I could remove overfitting by setting mtry to 1. However, having mtry at one does not minimise the MSE for the test_set and I have 29 features in the model meaning the default mtry should be 5. Therefore, I believe I must have some issues with my feature selections that is causing the overfitting. Either that or there are circumstances whereby mtry-1 is optimal? But I am not convinced by that.

Answer 1

While the bagging of random forests is meant to reduce overfitting, they generally will overfit more than e.g. a parametric model like logistic regression.

Having a larger gap between training and testing scores is not necessarily a problem; you may still prefer the model if its testing score is higher than other models'. In a purely statistical perfect world, the test score is all you should care about. In real problems, there are some other things to consider: a more-overfit tree model is likely less interpretable, potentially takes longer to train and make predictions, may be more susceptible to concept drift, etc.

You should, as you have, see if reducing the model's capacity can increase your testing score (and maybe see if doing so can reduce the train-test gap even if it costs a little on test score, see above). With min.node.size, I would suggest using a fraction of the training set size rather than the small fixed integers that are default in ranger. Note that ranger does also have a max.depth parameter if you'd prefer to fix that directly. Reducing mtry is also a good approach, but 1 strikes me personally as too random, and I'd worry that you'd need a ton of trees to stabilize results; using mtry in combination with a tree complexity parameter hopefully produces a higher test score without too outrageous of a train-test gap.