I am constructing what is known as an 'Expected Goals' model for football. This metric measures shot quality and a probability is assigned to a shot to achieve this, i.e. the chance a shot will be converted. To create this model I am using a random forest classifier. For evaluation purposes I am only interested in the accuracy of the probabilities rather than strictly classifying shots, therefore, I use the predictions to calculate the Mean Square Error where goal = 1 and no goal = 0. The MSE for the test set along with two benchmarks are as follows:
 "test.rf_mse: 0.0856633533734135"  "comparison_model_mse: 0.0820007160001345"  "naive_baseline_mse: 0.0912291249164997"
Note that the comparison model should be better than mine but the naive baseline should be worse. When looking at this the model looks to be doing okay. However, when I apply the same steps to the training set I get the following:
 "test.rf_mse: 0.0112001023587005"  "comparison_model_mse: 0.0722459417565357"  "naive_baseline_mse: 0.0858344459279039"
Here the MSE falls to unrealistic levels. Doesn't this mean that my model is overfitting? I understand that the idea that random forests can't overfit isn't strictly correct as all models can overfit to some extent, but for the model to be overfitting by this much must mean I am misunderstanding something here.
#FINAL MODEL FOR SHOTS DATASET set.seed(5555) trainIndex <- createDataPartition(shots$goal.miss, p = .75, list = FALSE) train_set <- shots[ trainIndex,] test_set <- shots[-trainIndex,] set.seed(1000) rf.shots <- randomForest(as.factor(goal.miss) ~ ., data=train_set, ntree=500, mtry=5) ###TEST RESULTS pred <- predict(rf.shots, test_set, type="prob") test_set$predictions.test <- pred[,2] mean(((test_set$goal.miss - test_set$predictions.test)^2)) ###TRAIN RESULTS pred <- predict(rf.shots, train_set, type="prob") train_set$predictions.train <- pred[,2] mean(((train_set$goal.miss - train_set$predictions.train)^2))
NOTE: I moved on to using the 'ranger' random forest package which allowed me to tune more hyperparameters. From a grid search, I found the only parameter that appeared to cause overfitting was mtry. I could remove overfitting by setting mtry to 1. However, having mtry at one does not minimise the MSE for the test_set and I have 29 features in the model meaning the default mtry should be 5. Therefore, I believe I must have some issues with my feature selections that is causing the overfitting. Either that or there are circumstances whereby mtry-1 is optimal? But I am not convinced by that.