2
$\begingroup$

LSTM can be used to generate text, can they be used to fix corrupted text files?

Say that my original was:

Alice was beginning to get very tired of sitting by her sister on the
bank, and of having nothing to do

Now it has become:

Alice0004 XGFT was beginning to get 900io ALPHA very tired 00New York 333 of siing XXXYYY by her siter on THIS IS 898 the
bank, and of &*& HHH GTY DOG AND CAT having nothing to do.

Basically, a lot of crap was added (some meaningless words but also meaningful words, and some letters may have been erased).

Is there any way LSTM would be able to fix this? (Maybe by cleaning some beforehand and then let LSTM do the rest)

$\endgroup$
1
$\begingroup$

You should use a Seq2Seq (which uses LSTM/GRU/RNNs) architecture for the task of cleaning text. The encoder network would take in the "noisy" sequence and the decoder would generate the "cleaned" sequence.

Seq2Seq models is often used for Neural Machine Translation (NMT).

The next thing is you will need a very large data set. Fortunately this shouldn't be too hard to generate noise injections (of real words, symbols, fake words, letter omissions etc) into sequences of text.

A good guide for a similar task is in the following post. The implementation uses Keras and has all of the code. https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8

$\endgroup$
0
$\begingroup$

Yes, you can definitely try that. I suggest you try Character level LSTM as it helps in fixing certain spelling mistakes. Even if not that, you can try a simple trie. LSTM would probably need a lot of data so I suggest maybe using some basic rules first as there are a lot of unknown words. Maybe it is worth using a keyword based rule to eliminate the meaningless words. Maybe use some Wordnet and get words in English dictionary to get existing words. Then, check if a word is very close to a dictionary word using Levenshtein distance or another metric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.