2
$\begingroup$

Most problem with machine learning projects I have faced is the lack of data. The samples available are enough to disqualify rule based approach but not enough for a neural network to train.

For example, to train a neural network (or even fine tune a pretrained model) on a new entity in an NER system takes a thousands of different records. And the requirement of these thousands of records is to have enough variations to avoid overfitting.

Generally as human we can detect patterns by carefully observing the data, however it becomes humanly not possible to detect all patterns in inputs, and that is where deep learning comes into play of automatically detecting the patterns to make a hypothesis.

Now my question is, what are possible ways, using which limited data can be used to train a neural network with limited data. Let me add some inputs from my side, which I believe are not sufficient:

  • Data Augmentation : for images rotating, scaling and skewing. For textual data, repeating text with some masking and embedding/synonym replacement.
  • what else?
$\endgroup$
  • 1
    $\begingroup$ In general it's a bad idea to use data augmentation techniques with text data, because it cannot properly represent real language: if the original data is modified only slightly then it's reasonably good quality but it doesn't add any coverage about general language; if it's modified more strongly then it adds unrealistic instances which don't correspond to real language. $\endgroup$ – Erwan Jul 18 '20 at 12:10
  • $\begingroup$ For the record, traditional ML methods are often much more efficient than DL in terms of quality / amount of training data ratio. But it's more work for finding the right features. $\endgroup$ – Erwan Jul 18 '20 at 12:14
  • $\begingroup$ I agree that augmenting text data is bad idea. That's why I posted this question. As far as traditional ML is considered, they can perform better with less data, but it needs extensive feature engineering and are limited at providing a threshold accuracy. $\endgroup$ – Sandeep Bhutani Jul 18 '20 at 13:05
2
$\begingroup$

As rightly pointed out by @erwan, it is a bad idea to use data augmentation with 'text data'

The problem of 'training with less data' can be approached in many ways, here I enlist two ways which helped me with significant impact:

(a) One approach would be to use semi-supervised approach. There are open sourced language models trained on insanely massive datasets, that can be used to perform a specific task like custom NER or sentence classification. Transfer learning is more useful today then ever.

(b) Anyways if we want to go ahead with Data augmentation, 'Sentence‐Chain Based Seq2seq Model for Corpus Expansion' is one of the proven methods to proceed. Please find the link to paper here.

I will strongly recommend to experiment with BERT before moving to Corpus Expansion. For introduction to transfer learning, BERT etc. please visit here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.