Language modelling for Spell Checker

Question

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use

1. Word-Level modelling
2. Character-Level modelling

plus I am preferring neural networks over Peter Norvig or N-gram/K-gram or any other "Vanilla" algorithm so that networks capture context

Example: Input: I have aplied fo r credit cart Output: I have applied for credit card

I have gone through models which use character level BLSTM with noise makers/error models and attention mechanism but not satisfied by result. https://towardsdatascience.com/creating-a-spell-checker-with-tensorflow-d35b23939f60

One of the article I have gone through is to use OpenNMT, but results were disastrous. https://medium.com/scribd-data-science-engineering/neural-spelling-corrections-and-the-importance-of-accuracy-977c0063d20f

Answer 1

A simple way to start with word-level:

Given a phrase credit cart, you'd like to correct this to credit card. Take all the correctly spelled words (assume for now that anything found in your dictionary is correctly spelled even if grammatically incorrect) and embed them into vectors using your favorite embedding, or better yet, fine-tuning that embedding on a corpus related to yours.

Next, for all the misspelled words, generate a small number of candidates based on edit distance. Then choose the winner from this set based on the distance from the mean of the embeddings from the correctly spelled set.

This certainly isn't optimal, but would be a framework to get started.

A way to start with word-level (sequence modeling):

Get your favorite word-level sequence model, fine tune it on corpus of your choosing.

Then, you process the document to be checked one word at time, and if word is misspelled, predict the next word based on the context before it. You should peel back the last layer to see the probabilities, then select the top N of those words most likely.

Then score the predicted next words based on edit distance to the actual next word, and choose that.

Final notes

For both these schemes you'd certainly have to do some engineering work and may need to weight by relevance or likelihood of the word, and so on.

Finally, I suspect the truly optimal way to do this is just to have a giant corpus of spelling mistakes & subsequent corrections (something a MS or a Google would have from their document software usage databases). With that much data, you might just be able to look up the most common misspelling and replace it, no ML needed. Or at very least narrow the candidate set of "correct" replacements for another simple scoring technique.

Otherwise, you could try with a character-level model to do something similar to my second example and move forward one character at a time, and for those misspelled words, generating a distribution of probabilities per character. Based on this you could cleverly try to figure out the best "path" that moves through your likely character distributions and again score the resultant candidates with some type of embedding distance to the words in the vicinity. Thus getting you the most likely character combination to replace the misspelled word with. It's not a great solution since you're not taking into account the words after, but perhaps with a bidirectional model you might be able to do this.

Anyway, hope this long ramble was somewhat helpful! Remember, a lot of deep learning for real use cases is just engineering!