Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE

Question

I have a follow-up question regarding this topic.

I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms.

My dataset has a minority class of successes which I would like to upsample using SMOTE or ADASYN.

I understand that the reasoning mentioned in this post applies to SMOTE and random upsampling by duplicating but does this also apply to upsampling via ADASYN? As I under ADASYN introduces even more randomness to the synthetic new observations so perhaps the correlation might be lower? In other words, does the use of ADASYN justify upsampling before the split or even upsampling the training and testing data separately?

I have seen a research paper that first applied the train test split and then upsampled the minority class using ADASYN in the testing and training dataset separately. This approach made better sense to me since as compared to upsampling before the train test split which introduces the possibility of leakage from the training to the test data this approach instead removes that possibility of leakage by separately upsampling the training and testing dataset. I have heard that this approach is also not fully correct since the testing dataset is supposed to replicate the real world and hence we are not supposed to change it in any way.

On the other hand instead of upsampling the minority class in the test dataset, I can even downsample the majority class which might be a better approach since the testing dataset remains still has only observations from the real world. Here we just gave the algorithm a fair chance (50:50) to pick between each class (1 or 0). Although once again the real world most likely will not have the 1s and 0s in equal proportions.

Additionally, some places also suggest doing the train test split proportionally so the training and testing datasets have the minority and majority class in equal proportions. As I understand this can be done using stratify=y when running the code. Please let me know if I need to do this and why?

Answer 1

There are a couple of questions you asked, but all of them are properly answered by one statement - your test data needs to reflect the real world as much as possible.

That is why you stratify when splitting data into a testing and training sets, your test data needs to reflect the reality as good as possible. You usually stratify over the target variable and some time or spatial variables, if the size of the original data allows for further stratification after the strata built on the target variable. When I worked with credit risk analysis, I stratified on the target variable and on a variable that measured which proportion of the investments the company has spent on permanent assets since it yielded better results than stratifying on the column that told me in which year the company got the loan granted and the dataset was not big enough to stratify on both the permanent assets and the year.

By upsampling the testing data, you are creating synthetic units, ones that may not ever occur "in the wild", some that are not even possible in the real world. You would upsample after the split, but only on the training data. You want to check how the upsampling affects the performance of your model on the data that is as objective as possible. I would like to check the paper you said used upsampling on both the train and test sets, because this is rather inappropriate unless the variables follow some distinct properties regarding their correlation, distribution and such that allows for such conclusions.

You noticed that both upsampling and downsampling have drawbacks. You want to compare their performance on the dataset in question an problem you want to solve. What I noticed data scientists and researches usually do is combine the upsampling and downsampling, but this might not work on your example and, depending on the complexity of patterns you want to find in the data, might be even worse than using only one of the methods.