Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not sure why the models tank if I don't do these things
I am attempting a multi-label classification problem. When I run RandomForest on the data without shuffling the results are quite good, in some cases suspiciously so.
Example:
feat1 feat2 ... label
chrom1 10 ... [a, b]
chrom1 200 ... [b]
... ... ... ...
... ... ... ...
chrom20 30 ... [c]
Notably two of the labels, b and c are completely sorted. All samples labeled b are at the beginning of the dataset and all samples labeled c are at the end. b and c are mutually exclusive. Additionally, the data is ordered by two of the features (feat1 and feat2 in the example).
Without shuffling the data, labels b and c are predicted nearly perfectly and the other labels are reasonably good.
After shuffling the data the performance for all labels decreases significantly.
I assume that information is leaking somehow due to the order of the data. However, I can't figure out how. The data is stratified when split, so the ratios of labels are close to equal in the train and test sets.
Anyone run into a similar issue before or see where I'm going wrong?
Note: this is not time-series data. Also I shuffle the data after labeling and before splitting into train/test, normalizing, imputing, etc.
Note 2: Somehow after killing the jupyter notebook kernel and running through the pipeline again, this time shuffling before labeling the data, the performance is now good again. I really don't understand whats going on. Is random forest not starting from scratch somehow?
