Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not sure why the models tank if I don't do these things

I am attempting a multi-label classification problem. When I run RandomForest on the data without shuffling the results are quite good, in some cases suspiciously so.


feat1    feat2 ...  label
chrom1   10    ...  [a, b]
chrom1   200   ...  [b]
...      ...   ...  ...
...      ...   ...  ...
chrom20  30    ...  [c]

Notably two of the labels, b and c are completely sorted. All samples labeled b are at the beginning of the dataset and all samples labeled c are at the end. b and c are mutually exclusive. Additionally, the data is ordered by two of the features (feat1 and feat2 in the example).

Without shuffling the data, labels b and c are predicted nearly perfectly and the other labels are reasonably good.

After shuffling the data the performance for all labels decreases significantly.

I assume that information is leaking somehow due to the order of the data. However, I can't figure out how. The data is stratified when split, so the ratios of labels are close to equal in the train and test sets.

Anyone run into a similar issue before or see where I'm going wrong?

Note: this is not time-series data. Also I shuffle the data after labeling and before splitting into train/test, normalizing, imputing, etc.

Note 2: Somehow after killing the jupyter notebook kernel and running through the pipeline again, this time shuffling before labeling the data, the performance is now good again. I really don't understand whats going on. Is random forest not starting from scratch somehow?

  • $\begingroup$ What does "shuffling before/after labeling" mean? What labeling? $\endgroup$
    – Ben Reiniger
    Sep 16, 2021 at 20:26
  • $\begingroup$ Assigning the classification labels to the samples. $\endgroup$
    – tensormoby
    Sep 18, 2021 at 1:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.