3
$\begingroup$

Problem statement

I have a text multi-label classification dataset, and I've found a problem with the dataset sampling.

I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.

The preprocessing step simply consists in transforming the labels into OneHot representation and keep only the N most frequent ones. I expect similar (the same) behaviour, but I'm getting really weird results. Let's take a closer look.

Train+Test all together and then split

Start with:

|ID   |TEXT  |LABELS|
|-----|------|------|
|1.txt|the   |A:B   |
|2.txt|lazy  |B     |
|3.txt|fox   |C     |
|4.txt|jumps |B:C   |
|5.txt|over  |C:D   |
|6.txt|crazy |D     |

After preprocessing and split: Train

|ID   |TEXT|A|B|C|D|
|-----|----|-|-|-|-|
|1.txt|the |1|1|0|0|
|2.txt|lazy|0|1|0|0|
|3.txt|fox |0|0|1|0|

Test

|ID   |TEXT |A|B|C|D|
|-----|-----|-|-|-|-|
|4.txt|jumps|0|1|1|0|
|5.txt|over |0|0|1|1|
|6.txt|crazy|0|0|0|1|

The results are good. Let's take this as reference. F1-Score = 0.61.

Pre-made Train/Test split

Start with:

Train

|ID   |TEXT|LABELS|
|-----|----|------|
|1.txt|the |A:B   |
|2.txt|lazy|B     |
|3.txt|fox |C     |

Test

|ID   |TEXT |LABELS|
|-----|-----|------|
|4.txt|jumps|B:C   |
|5.txt|over |C:D   |
|6.txt|crazy|D     |

After preprocessing and split:

Train

|ID   |TEXT|A|B|C|
|-----|----|-|-|-|
|1.txt|the |1|1|0|
|2.txt|lazy|0|1|0|
|3.txt|fox |0|0|1|

Test

|ID   |TEXT |B|C|D|
|-----|-----|-|-|-|
|4.txt|jumps|1|1|0|
|5.txt|over |0|1|1|
|6.txt|crazy|0|0|1|

The results are totally degradated. F1-Score = 0.15.

What is going on? What could be causing the divergence in results?

Extra information

  • The labels predicted at the prediction step are not compatible with the labels on the test set. I've taken that into account and is correctly managed, that's not the problem.
  • The splits are exactly the same. The documents in train/test are the same in both situations.
$\endgroup$
4
  • $\begingroup$ The great degradation (F1-Score from 0.6 to 0.15) was caused by a bug in code. Anyway, it's true that having incompatible train and test sets causes degradation (the more divergence, the more degradation, of course), that's why I have accepted @shadowtalker answer. Thanks! $\endgroup$ – Alber8295 Oct 8 '18 at 10:34
  • $\begingroup$ Is your dataset related to news corpus? I am currently in search of multilabeled english news corpus dataset.could you please share the link? $\endgroup$ – Annonymous programmer May 31 '19 at 12:11
  • $\begingroup$ @Alber8295...I am looking for a english news multi labeled corpus...Can you help me? $\endgroup$ – Annonymous programmer Jun 1 '19 at 3:54
  • $\begingroup$ Could you please share with me your email address? I don't use stackoverflow frequently so I am not aware about messaging in stackoverflow...... $\endgroup$ – Annonymous programmer Jun 3 '19 at 10:25
1
$\begingroup$

I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.

Strategy 2 is "correct" from a "don't reuse your data" perspective. Data pre-processing is part of training. Your train/test pipeline should, in principle, account for this. For example, if you have a feature $A$ with 5 categories, but only 4 of those categories appear in the data, the correct thing to do is to only use 4 categories in your one-hot encoding, and treat the 5th category as an "unknown" value.

In practice, however, this isn't always practical. In some cases the pre-processing pipeline is compute-intensive (e.g. training word vectors) or requires processing a massive amount of data (e.g. ingesting 5 GB of images off of AWS). In those cases, you have to accept the fact that some knowledge of your test set will "leak" into your training set by way of your preprocessing steps, but it's usually not so bad.

Other times you don't have enough data after train/test splitting to conduct preprocessing correctly. Or you didn't stratify your splits, and so you ended up with incomplete class representation, or and otherwise imbalanced training set. This is where you find yourself. Just look at the headings to see why:

Train:

|ID   |TEXT |A|B|C|

Test:

|ID   |TEXT |B|C|D|

Class "A" is missing from your test set, and class "D" is missing from your training set. This will ruin just about any model.

There has been some research done on the stratification of multi-label data. Notably, there is "On the Stratification of Multi-label Data" (Sechidis, Tsoumakas, and Vlahavas; ECML PKDD 2011), which introduces "iterative stratification". There is an implementation of this technique in the scikit-multilearn Python library.

$\endgroup$
3
  • $\begingroup$ I'm aware that train label set and test label set don't match, and of course this is bad. I am working with 10 classes, train set has 1 class that is not on test and test has 1 class that is not on train, so the intersection is 9 classes out of 10. Could this degradate results from F-Score 0.6 to 0.15? Sounds strange to me, too much degradation, isn't it? $\endgroup$ – Alber8295 Oct 7 '18 at 19:00
  • $\begingroup$ Unless the omitted class is very rare, then literally every one of those records will be misclassified. So it's possible, depending on your data. Hard to comment further without more information. $\endgroup$ – shadowtalker Oct 7 '18 at 20:44
  • 1
    $\begingroup$ I've found a bug that was causing that big degradation. Without the bug, there is degradation, as you stated, but much smaller. So, your answer was the correct one. Marked as accepted answer. Thanks! $\endgroup$ – Alber8295 Oct 8 '18 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.