# Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not sure why the models tank if I don't do these things

I am attempting a multi-label classification problem. When I run RandomForest on the data without shuffling the results are quite good, in some cases suspiciously so.

Example:

feat1    feat2 ...  label
chrom1   10    ...  [a, b]
chrom1   200   ...  [b]
...      ...   ...  ...
...      ...   ...  ...
chrom20  30    ...  [c]


Notably two of the labels, b and c are completely sorted. All samples labeled b are at the beginning of the dataset and all samples labeled c are at the end. b and c are mutually exclusive. Additionally, the data is ordered by two of the features (feat1 and feat2 in the example).

Without shuffling the data, labels b and c are predicted nearly perfectly and the other labels are reasonably good.

After shuffling the data the performance for all labels decreases significantly.

I assume that information is leaking somehow due to the order of the data. However, I can't figure out how. The data is stratified when split, so the ratios of labels are close to equal in the train and test sets.

Anyone run into a similar issue before or see where I'm going wrong?

Note: this is not time-series data. Also I shuffle the data after labeling and before splitting into train/test, normalizing, imputing, etc.

Note 2: Somehow after killing the jupyter notebook kernel and running through the pipeline again, this time shuffling before labeling the data, the performance is now good again. I really don't understand whats going on. Is random forest not starting from scratch somehow?

• What does "shuffling before/after labeling" mean? What labeling? Sep 16, 2021 at 20:26
• Assigning the classification labels to the samples. Sep 18, 2021 at 1:22