How to do spelling correction for a language but also correct some words in other language

Question

I want to do spell correction for the portuguese language, specifically for restaurant bots. The problem is that food names aren't always in portuguese as well, and for that reason are the most likely to need correction as the user would not always know how to spell it correctly.

I though of a few things. For example:

• Training the model with portugese words + food words spelled wrong, but it'd be very difficult to find these food words
• Training the model for several different languages (but I guess would make it confused and actually correct a lot of things wrong).
• Training the model with portuguese words and for the food words, use something that picks words that the user wrote and try to approximate them to the list of food words. (woudln't it be slow?)

What would be a good solution to this problem that can be fast to be used with lots of requests?

Answer 1

There are many ways to build a spell corrector. One of the simplest is:

1. Detect an incorrect word
2. Generate candidate suggestions
3. Score and rank the candidate replacements

For detecting an incorrect word, a simplifying assumption is that any word not a dictionary is a spelling error. Otherwise, you have to build a separate model to detect if a word is a potential spelling mistake based the current context.

To generate candidate words, you need to find dictionary words that are similar to the incorrect word. That requires defining "similar", typically similar is measured by edit distance (e.g., deletes, transposes, replaces, or inserts of individual characters).

For scoring and ranking, probabilities of candidate words are weighted by a language model and channel model. A language model weights how likely a word will appear in the current context. A channel model reflects if an error happens depending on how the word is transmitted (e.g., full computer keyboard errors are different from mobile phone errors).

For your example, you need to construct a custom dictionary of all possible words and a corpus reflecting the frequency of occurrences for the custom dictionary words. If possible, construct an error model of common mistakes.

Peter Norvig goes into greater detail here.

Later you can add Deep Learning methods, but it is better to start with a simpler approach that requires less data.

Answer 2

You could solve the problem in an unsupervised approach.

1. build your data distribution with all food Portuguese words
2. formulate your data points as a sequence of characters, rnn-effectiveness
3. transform those words into its phonemic form, see text to phonemes converter
4. use denoising auto-encoder, instead of using random gaussian as your corruption process.
   Since the problem is discrete, you could use some heuristics to corrupt each word by replacing random phonemes with others which is most likely to be similar.

You have to find some automated way to do that corruption, get some insights from customers history data and see their words spellings error.

As for the architecture, vanilla RNN is fine. see more Creating a Spell Checker with TensorFlow

In a nutshell: it's formulated as a seq2seq problem, the source is the misspelled word, while the target is the correct one. DAE will be trained on input that is corrupted stochastically, thus it must learn to guess the distribution of the missing information (reconstruct the clean original input)

Answer 3

You could use menus in other languages as part of your corpus along with your Portuguese words. I would focus on having text that is correctly spelled for the corpus. Then you can use something like Levenshtein distance to give candidates for words never seen in your corpus (implying incorrectly spelled or missing).