I have a large text corpus (i.e. 30 million sentences, all in lowercase in the format of Penn Treebank) that I want to use to train a neural network for natural language generation. What preprocessing steps would you recommend here? The sentences originate from formal text (i.e. books). I plan to use named entity recognition in order to replace named entities such as people, locations, and organisations during training and generation, and adding them back in for the final output. Any other suggestions?
$\begingroup$
$\endgroup$
9
-
$\begingroup$ What was the motivation to remove entities? Is it related to how you will use the generated text? It both makes things easier (a much smaller vocab) and harder (dependency on your named entity code, and having to deal with abstract output). E.g. your model (in the ideal case) generates "PERSON slapped PERSON round the face, smiled to PERSON, then jumped on the train to PLACE." I don't know if there are two or three people in that scene; also most of the time one or two or even all three would have been pronouns. $\endgroup$– Darren CookFeb 5 at 10:07
-
$\begingroup$ I have a lot of different named entities in my sentences. In the example of PERSON, that means the model will have difficulties learning the association between a name and the rest of the sentence when it's always a different name. The problem you are raising can be dealt with by using tokens like "PERSON1", "PERSON2" etc and randomly substituting them in for names. Same for places and such. So "Daniel slapped Javier round the face, smiled to Léa, then jumped on the train to Rome" -> "PERSON2 slapped PERSON1 round the face, smiled to PERSON3, then jumped on the train to PLACE1" $\endgroup$– postnubilaphoebusFeb 6 at 10:07
-
1$\begingroup$ Re "Daniel slapped Javier round the face, smiled to Léa, then jumped on the train to Rome" be turned into PERSON1/2/3. This is what I meant by the challenge with pronouns. It is just as likely to be "Bob slapped Alex, smiled at her, then jumped on their train." (I chose "Alex" to make the "her" ambiguous: if Alex is female "her" is PERSON2, if Alex is male then "her" is PERSON3, and "their" could be referring to anyone at this point!) The thing about transformers, as Noe notes, is that they handle this kind of thing better than hand-built heuristics. $\endgroup$– Darren CookFeb 6 at 10:48
-
2$\begingroup$ @postnubilaphoebus I think your Idea is similar to what the authors of the article Adversarial Text Generation Without Reinforcement Learning propose. At some point, I was interested in text generation with GANs, but never saw them work better than LMs. Here's a survey on the topic with a useful summary table. $\endgroup$– noeFeb 6 at 13:36
-
1$\begingroup$ Re "I don't see the problem yet when I don't touch pronouns" If you sometimes train with PERSON1/2/3, and sometimes with PERSON1 and a "he" and a "she", the model will learn to generate all these forms. Whereas your ideal is have it generate only the version with PERSON1/2/3 in it, and then you can choose which of those to change into pronouns. If your model generates the pronouns then half the time it will guess wrong. $\endgroup$– Darren CookFeb 6 at 14:25
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
Some comments:
- With Transformers and subword vocabularies (e.g. byte-pair encoding (BPE)), usually there is no need to remove named entities because the model learns to handle them just fine. For instance, in machine translation models learn to copy them verbatim or to translate them without much problem. My advice would be not to overcomplicate things unless proven necessary.
- Again, with Transformers and BPE usually there is no need for much preprocessing. If any, I would ensure there is no garbage in your data. What has worked for me in the past is to sort the sentences and eyeball the first and last sentences, where you can usually find the garbage, and remove them manually.
