Data augmentation techniques for image data and audio data (eg speech recognition) have proven successful and are now common.

Are there libraries or techniques for augmenting text data?

For example:

in: 'How are you?'
out: ['how are you?', 'HOW ARE YOU?', 'hwo are y ou?', 'How're you?', 'how r u', ...]

  • Can't you simply randomize a word insertion from a word list? "Pizza are you?" – xtian Oct 30 '16 at 1:23
  • 1
    No, because ideally noise should be realistic. Typos and spelling errors are not random, they are functions of homonymy, word length, char cluster freqs, keyboard layouts, mobile input... – A. M. Bittlingmayer Oct 30 '16 at 7:45
  • 1
    Could you specify what you hope to gain? If it is more data you want, you might want to use SMOTE. If you hope train a model that can handle spelling errors, you might want to sanitize your tokens against a corpus using something like Hamming distance. – S van Balen Jan 27 '17 at 11:27
  • @SvanBalen Thanks for the suggestion. The goal: gain realistic data, given cleanish data. SMOTE is interesting, but let us assume imbalance is not a problem. The transformations should be such that the output labels -- eg POS tags, sentiment, translation quality -- can be safely kept. – A. M. Bittlingmayer Jan 27 '17 at 11:50
  • I don't know any library that introduces spelling errors, but it is an interesting idea. The most obvious answer would be to correct the errors in the input once you are in the application fase, in the same way they have been corrected in your training data, or correct them automatically. But given your question, you could take some of the typo-models (such as Hamming, but there are several aimed at mobile usage fi) and, gauge the typical typo rate and write a filter that applies those models in opposite direction to the point of your observed error rate. – S van Balen Jan 27 '17 at 12:09
up vote 2 down vote accepted

I you want some kind of data-sets like Google spell checking data I suggest you look into the The WikEd Error Corpus dataset. The corpus consists of more than 12 million sentences with a total of 14 million edits of various types, this edits include: spelling error corrections, grammatical error corrections, stylistic changes. All these from the Wikipedia correction history. The owners (authors) of the data-set describe the data mining process in this paper. Also check this question in quora it contains links to various data-sets with spelling errors. Finally this page can also be useful.

You can code certain simple rules like the ones you have mentioned in the question. Additionally, you can use knowledge bases like Freebase and WordNet to enrich your language model. Note that this will not necessarily "noisify" your data but would have effect similar to the effect on data augmentation on say images for downstream tasks.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.