I have a text multi-label classification dataset, and I've found a problem with the dataset sampling.
I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.
The preprocessing step simply consists in transforming the labels into OneHot representation and keep only the N most frequent ones. I expect similar (the same) behaviour, but I'm getting really weird results. Let's take a closer look.
Train+Test all together and then split
|ID |TEXT |LABELS| |-----|------|------| |1.txt|the |A:B | |2.txt|lazy |B | |3.txt|fox |C | |4.txt|jumps |B:C | |5.txt|over |C:D | |6.txt|crazy |D |
After preprocessing and split: Train
|ID |TEXT|A|B|C|D| |-----|----|-|-|-|-| |1.txt|the |1|1|0|0| |2.txt|lazy|0|1|0|0| |3.txt|fox |0|0|1|0|
|ID |TEXT |A|B|C|D| |-----|-----|-|-|-|-| |4.txt|jumps|0|1|1|0| |5.txt|over |0|0|1|1| |6.txt|crazy|0|0|0|1|
The results are good. Let's take this as reference.
F1-Score = 0.61.
Pre-made Train/Test split
|ID |TEXT|LABELS| |-----|----|------| |1.txt|the |A:B | |2.txt|lazy|B | |3.txt|fox |C |
|ID |TEXT |LABELS| |-----|-----|------| |4.txt|jumps|B:C | |5.txt|over |C:D | |6.txt|crazy|D |
After preprocessing and split:
|ID |TEXT|A|B|C| |-----|----|-|-|-| |1.txt|the |1|1|0| |2.txt|lazy|0|1|0| |3.txt|fox |0|0|1|
|ID |TEXT |B|C|D| |-----|-----|-|-|-| |4.txt|jumps|1|1|0| |5.txt|over |0|1|1| |6.txt|crazy|0|0|1|
The results are totally degradated.
F1-Score = 0.15.
What is going on? What could be causing the divergence in results?
- The labels predicted at the prediction step are not compatible with the labels on the test set. I've taken that into account and is correctly managed, that's not the problem.
- The splits are exactly the same. The documents in train/test are the same in both situations.