Random Forest Overfitting, issues with mtry=1?

I am constructing what is known as an 'Expected Goals' model for football. This metric measures shot quality and a probability is assigned to a shot to achieve this, i.e. the chance a shot will be converted. To create this model I am using a random forest classifier. For evaluation purposes I am only interested in the accuracy of the probabilities rather than strictly classifying shots, therefore, I use the predictions to calculate the Mean Square Error where goal = 1 and no goal = 0. The MSE for the test set along with two benchmarks are as follows:

[1] "test.rf_mse: 0.0856633533734135"
[1] "comparison_model_mse: 0.0820007160001345"
[1] "naive_baseline_mse: 0.0912291249164997"


Note that the comparison model should be better than mine but the naive baseline should be worse. When looking at this the model looks to be doing okay. However, when I apply the same steps to the training set I get the following:

[1] "test.rf_mse: 0.0112001023587005"
[1] "comparison_model_mse: 0.0722459417565357"
[1] "naive_baseline_mse: 0.0858344459279039"


Here the MSE falls to unrealistic levels. Doesn't this mean that my model is overfitting? I understand that the idea that random forests can't overfit isn't strictly correct as all models can overfit to some extent, but for the model to be overfitting by this much must mean I am misunderstanding something here.

#FINAL MODEL FOR SHOTS DATASET
set.seed(5555)
trainIndex <- createDataPartition(shots$goal.miss, p = .75, list = FALSE) train_set <- shots[ trainIndex,] test_set <- shots[-trainIndex,] set.seed(1000) rf.shots <- randomForest(as.factor(goal.miss) ~ ., data=train_set, ntree=500, mtry=5) ###TEST RESULTS pred <- predict(rf.shots, test_set, type="prob") test_set$$predictions.test <- pred[,2] mean(((test_set$$goal.miss - test_set$predictions.test)^2))

###TRAIN RESULTS
pred <- predict(rf.shots, train_set, type="prob")
train_set$$predictions.train <- pred[,2] mean(((train_set$$goal.miss - train_set$predictions.train)^2))  NOTE: I moved on to using the 'ranger' random forest package which allowed me to tune more hyperparameters. From a grid search, I found the only parameter that appeared to cause overfitting was mtry. I could remove overfitting by setting mtry to 1. However, having mtry at one does not minimise the MSE for the test_set and I have 29 features in the model meaning the default mtry should be 5. Therefore, I believe I must have some issues with my feature selections that is causing the overfitting. Either that or there are circumstances whereby mtry-1 is optimal? But I am not convinced by that. • It's a bit of a guess since I don't know R but is the very last line of your code mean(((test_set$goal.miss - test_set\$predictions.train)^2)) really correct? Nov 26, 2019 at 22:49
• Sorry, yes that was a typo. I was trying to make the code look clearer on here but messed up. I've correct the code to how it should be. I have moved onto the 'ranger' package which is a bit smarter and allows me to tune for hyperparameters. I found that the only parameter that appeared to cause overfitting was mtry and I could remove the overfitting by setting mtry = 1. However, I have 29 features so the default would be 5, so I think I have some issues with my feature selection Nov 27, 2019 at 1:09
• What is the "naive baseline" model? Nov 27, 2019 at 7:35
• The naive baseline model simply assigns a probability of 10% to every shot. Nov 27, 2019 at 10:11
• Reducing the tree depth doesn't help? Nov 27, 2019 at 16:39

You should, as you have, see if reducing the model's capacity can increase your testing score (and maybe see if doing so can reduce the train-test gap even if it costs a little on test score, see above). With min.node.size, I would suggest using a fraction of the training set size rather than the small fixed integers that are default in ranger. Note that ranger does also have a max.depth parameter if you'd prefer to fix that directly. Reducing mtry is also a good approach, but 1 strikes me personally as too random, and I'd worry that you'd need a ton of trees to stabilize results; using mtry in combination with a tree complexity parameter hopefully produces a higher test score without too outrageous of a train-test gap.