3
$\begingroup$

Data augmentation techniques for image data and audio data (eg speech recognition) have proven successful and are now common.

Are there libraries or techniques for augmenting text data?

For example:

in: 'How are you?'
out: ['how are you?', 'HOW ARE YOU?', 'hwo are y ou?', 'How're you?', 'how r u', ...]

Improve this question
$\endgroup$
6
  • $\begingroup$ Can't you simply randomize a word insertion from a word list? "Pizza are you?" $\endgroup$
    – xtian
    Commented Oct 30, 2016 at 1:23
  • 1
    $\begingroup$ No, because ideally noise should be realistic. Typos and spelling errors are not random, they are functions of homonymy, word length, char cluster freqs, keyboard layouts, mobile input... $\endgroup$
    – Adam Bittlingmayer
    Commented Oct 30, 2016 at 7:45
  • 1
    $\begingroup$ Could you specify what you hope to gain? If it is more data you want, you might want to use SMOTE. If you hope train a model that can handle spelling errors, you might want to sanitize your tokens against a corpus using something like Hamming distance. $\endgroup$
    – S van Balen
    Commented Jan 27, 2017 at 11:27
  • $\begingroup$ @SvanBalen Thanks for the suggestion. The goal: gain realistic data, given cleanish data. SMOTE is interesting, but let us assume imbalance is not a problem. The transformations should be such that the output labels -- eg POS tags, sentiment, translation quality -- can be safely kept. $\endgroup$
    – Adam Bittlingmayer
    Commented Jan 27, 2017 at 11:50
  • $\begingroup$ I don't know any library that introduces spelling errors, but it is an interesting idea. The most obvious answer would be to correct the errors in the input once you are in the application fase, in the same way they have been corrected in your training data, or correct them automatically. But given your question, you could take some of the typo-models (such as Hamming, but there are several aimed at mobile usage fi) and, gauge the typical typo rate and write a filter that applies those models in opposite direction to the point of your observed error rate. $\endgroup$
    – S van Balen
    Commented Jan 27, 2017 at 12:09

3 Answers 3

Reset to default
3
$\begingroup$

I you want some kind of data-sets like Google spell checking data I suggest you look into the The WikEd Error Corpus dataset. The corpus consists of more than 12 million sentences with a total of 14 million edits of various types, this edits include: spelling error corrections, grammatical error corrections, stylistic changes. All these from the Wikipedia correction history. The owners (authors) of the data-set describe the data mining process in this paper. Also check this question in quora it contains links to various data-sets with spelling errors. Finally this page can also be useful.

Improve this answer
$\endgroup$
0
$\begingroup$

You can code certain simple rules like the ones you have mentioned in the question. Additionally, you can use knowledge bases like Freebase and WordNet to enrich your language model. Note that this will not necessarily "noisify" your data but would have effect similar to the effect on data augmentation on say images for downstream tasks.

Improve this answer
$\endgroup$
0
$\begingroup$

A student of mine did eventually end up doing exactly this:

https://noisemix.github.io/

data generation for natural language

 pip install noisemix

She showed that nosification brought significant improvements on tasks like classifications.

However, there is much more to do, and noise is often task- and domain-specific.

Improve this answer
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.