1
$\begingroup$

I've a text dataset with ~20000 samples (which is not enough). I used text augmentation to "invent" more samples so essentially I've multiplied each sample by 10 - ending up with ~200000 samples (each of the 10 is a different kind of augmentation method).

I did that for the whole dataset before splitting it to train and test datasets. Should I've done it only for the training dataset?

UPDATE:

based on the answer given, I've a follow up question:

What should the order of operations should be? I understood by now that there augmentation should be done only on the train dataset, but what about tokenization and stemming?

Is the below the correct order?

  1. splitting data set into 2 datasets: train and test
  2. perform augmentation only on train dataset
  3. stemming and tokenize both datasets
  4. text encoding
  5. create model
  6. fit data on model
  7. evaluate

I guess my question is regarding step #3. Is it correctly placed in the above order?

$\endgroup$

1 Answer 1

3
$\begingroup$

There are at least two reasons why the split should be made first:

  • In theory at least, there is a true distribution of the data for the target task. Any model should always be evaluated on the true distribution of the data, because the goal is to predict on this distribution. Since data augmentation modifies this distribution, it's as if the model is evaluated on a different task instead of the target task.
  • Augmentation techniques create artificial instances which are usually easier for a model to classify, since they follow the pattern used to generate them. If these easier instances are used in the test set, the performance is very likely to be overestimated. This is a case of data leakage: some information from the training set "leaks" in the test set.

Final remark: I think that augmentation techniques should be used with caution on text data. In general text is very difficult to simulate.

$\endgroup$
5
  • $\begingroup$ So I'm taking from your answer I should swap the order of operations (i.e. split before augmentation). Regarding your final remark: can you suggest any other technic to enlarge the dataset? $\endgroup$
    – Ben
    Dec 29, 2021 at 13:33
  • $\begingroup$ @Ben yes, for proper evaluation the test set should only contain original instances. In general I don't think there's any need to enlarge the dataset: if you obtain the best possible model with the available data, it's unlikely that data augmentation would increase performance (you can try and compare experimentally, of course). $\endgroup$
    – Erwan
    Dec 29, 2021 at 22:20
  • $\begingroup$ About your updated question: usually the preprocessing steps like tokenization and stemming should be performed in a function which receives any dataset, this way it's just a matter of calling this function independently for training and test data. This design has several advantages: (1) it's more adapted for any application of the trained model later, which is the goal in theory. (2) it would require less work to run for instance cross-validation, you can just call the function for the training set at every split. $\endgroup$
    – Erwan
    Dec 29, 2021 at 22:24
  • $\begingroup$ Thanks, but assuming I will use augmentation, should it be performed on a sanitized form of text (i.e. stemmed and tokinized) or it doesn't matter? $\endgroup$
    – Ben
    Dec 30, 2021 at 8:35
  • $\begingroup$ @Ben I'm not sure, it might depend on what kind of augmentation but I don't think it matters much. $\endgroup$
    – Erwan
    Dec 30, 2021 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.