Predicting Missing Word in Text

Question

I know about BERT and other solutions when you masking some words and try to predict them. But let say I have a text:

Transformer have taken the of Natural Processing by storm, transforming the field by leaps and bounds. New, bigger, and better models to crop up almost every , benchmarks in performance across a wide variety of tasks.

And I cannot in advance say to BERT where masking is. I am looking for an algorithm which can understand where missing words are and after that predict them.

Answer 1

It's not simple, but doable. I suggest you to create training data in the following way: take a text corpus, as large as possibile, and remove words sampled randomly. Then train an seq2seq RNN to map this "deteriorated" text with its original.

The RNN you need won't be too different from an NMT model, but it's goal is different of course.

It's the first time I encountered this kind of task, therefore I can't say what is the state of the art.