I have developed a Random Forest that gives varying results depending on the random state of the test train split. This is normal, because a lot of the values in the data are extreme, without being actual outliers. When saving the model for future use on unseen data, should I use the random_state that splits the data in a way resulting in best accuracy, recall, f1 OR should I use a random_state that gives average results?
2 Answers
If you only use random state
for test/train split then I don't think it matters for future inference. You don't even need that to run the inference.
If you use random state
on multiple place in the training pipeline, then it depends on your actual usage. For Random Forest, all the trees are fixed after training so there is no random factor in inference. Therefore, random state
is not important anymore.
If the model depends a lot on random_state, it is unstable, and that is not a good thing. The best would be to address that at the root, for example by increasing the amount or quality of the data.
Another alternative, to get better estimates of the generalized performance, would be to use multiple train/test splits (cross-validation), and pick the model that maximizes the cross-validation score (either mean/median or worst-case, etc).
Furthermore, better tuning of hyperparameters might improve the model stability (reduce sensitivity to random aspects of the data). For example restricting depth in your RandomForest.
-
1$\begingroup$ Yep, don't hide instability by setting the seed. Pre-processing the data (log_scaling), reducing the overfitting (regularisation), ensembling (using multiple models) might help. $\endgroup$ Nov 21, 2022 at 20:45
-
$\begingroup$ Each node in a Random Forest is not comparing feature values, it is splitting sorted lists. Log_scaling the data and regularization do not improve the model. $\endgroup$ Nov 25, 2022 at 9:33
-
$\begingroup$ Agree on the log-scaling. But regularization in terms of limiting depth/splits does tend to improve generalized performance of the model? $\endgroup$ Nov 25, 2022 at 10:33