Which random_state to use in test_train_split when deploying final model?

Question

I have developed a Random Forest that gives varying results depending on the random state of the test train split. This is normal, because a lot of the values in the data are extreme, without being actual outliers. When saving the model for future use on unseen data, should I use the random_state that splits the data in a way resulting in best accuracy, recall, f1 OR should I use a random_state that gives average results?

Answer 1

If you only use random state for test/train split then I don't think it matters for future inference. You don't even need that to run the inference.

If you use random state on multiple place in the training pipeline, then it depends on your actual usage. For Random Forest, all the trees are fixed after training so there is no random factor in inference. Therefore, random state is not important anymore.

Answer 2

If the model depends a lot on random_state, it is unstable, and that is not a good thing. The best would be to address that at the root, for example by increasing the amount or quality of the data.

Another alternative, to get better estimates of the generalized performance, would be to use multiple train/test splits (cross-validation), and pick the model that maximizes the cross-validation score (either mean/median or worst-case, etc).

Furthermore, better tuning of hyperparameters might improve the model stability (reduce sensitivity to random aspects of the data). For example restricting depth in your RandomForest.