Is stratified sampling necessary (random forest, Python)?

Question

I use Python to run a random forest model on my imbalanced dataset (the target variable was a binary class). When splitting the training and testing dataset, I struggled whether to used stratified sampling (like the code shown) or not. So far, I observed in my project that the stratified case would lead to a higher model performance. But I think if I will use my model to predict the new cases which would highly probably differ in the distribution of target class with my current dataset. So I inclined to loosen this constrain and use the unstratified split. Could anyone can advice to clarify this point?

train,test=train_test_split(myDataset, test_size=0.25, stratify=y)

Answer 1

If the number of values belonging to each class are unbalanced, using stratified sampling is a good thing. You are basically asking the model to take the training and test set such that the class proportion is same as of the whole dataset, which is the right thing to do. If your classes are balanced then a shuffle (no stratification needed here) can basically guarantee a fair test and train split.

Now your model will be capable or at least enough equipped to predict the outnumbered class (class with lesser points in number). That is why instead of just calculating Accuracy, you have been given other metrics like Sensitivity and Specificity. Keep a watch on these, these are the guardians.

Hope this helps.