I have a large text corpus (i.e. 30 million sentences, all in lowercase in the format of Penn Treebank) that I want to use to train a neural network for natural language generation. What preprocessing steps would you recommend here? The sentences originate from formal text (i.e. books). I plan to use named entity recognition in order to replace named entities such as people, locations, and organisations during training and generation, and adding them back in for the final output. Any other suggestions?
- With Transformers and subword vocabularies (e.g. byte-pair encoding (BPE)), usually there is no need to remove named entities because the model learns to handle them just fine. For instance, in machine translation models learn to copy them verbatim or to translate them without much problem. My advice would be not to overcomplicate things unless proven necessary.
- Again, with Transformers and BPE usually there is no need for much preprocessing. If any, I would ensure there is no garbage in your data. What has worked for me in the past is to sort the sentences and eyeball the first and last sentences, where you can usually find the garbage, and remove them manually.