0
$\begingroup$

Let's say I have a CSV file (single column) with a list of words as input. Meaning the file looks like below

Input Terms

Subaaaash                         
Paruuru                          
Mettromin                        
Paracetamol                      
Crocedinin                        
Vehiclehcle                      
Buildding                        

Dict terms #this dict has around million records and I have to find a closest match for input term from this dictionary

Metformin 250 MG
Metformin
.....
Crocin
Vehcile 
Paru
Subash

Now, I expect my output to be like as shown below

As you can see that red colored Paru is not a correct match for paracetamol, but that's the closest match we can get for the input term paracetamol from the dictionary. So, I would like to do matching based on

1) Word sounds (when pronounced). Phenotics

2) Spelling mistake corrections

3) Find the best matching word from the dictionary for input terms

enter image description here

Can you let me know how can I do the above?

$\endgroup$

1 Answer 1

1
$\begingroup$

So, your question concerns how to effectively translate the input words into their proposed correct words (e.g. Paruuuu --> Paru) via phonetics and spelling mistake corrections.

My first idea on this would be to use a deep sequence to sequence model. In a sequence to sequence model, we encode the input word (e.g. Paruuuu) as a sequence of characters into an encoder (effectively an RNN / LSTM, etc.) into a "hidden representation".
Then you decode your hidden representation as a sequence of phonemes (which denote how we pronounce the word) with a decoder (again, another RNN / LSTM, etc.).
Then, we can take this sequence of phonemes as input into another encoder and then decode with a neural network, where the output layer is a softmax layer, which computes the probability distribution over all words in your vocabulary (in you Dict terms) and select the word the highest probability.
So overall, we have:

Encode input word --> Decode output phoneme sequence --> Encode output phoneme sequence --> Decode with neural network to classify word.


The proposed method is supervised, so of course, you will need examples of input words and their correct "translations" (e.g. ("Paruuu", "Paru"))


Here is an article which gives a good intuition behind sequence to sequence models which can classify your input words: https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346

$\endgroup$
6
  • $\begingroup$ Hi @Shepan6, thanks for your response. Upvoted $\endgroup$
    – The Great
    Jun 12, 2020 at 8:37
  • $\begingroup$ In addition, is there anyway to do this without ML? Meaning through some string comparison approach (leivenstein distance/edit distance etc) $\endgroup$
    – The Great
    Jun 12, 2020 at 8:39
  • $\begingroup$ Because I have a huge dictionary containing millions of records... But my input records are only in 1000's... So is there anyway to use dictionary items to get the right spelling for input terms without using ML?... $\endgroup$
    – The Great
    Jun 12, 2020 at 8:40
  • $\begingroup$ Yes @TheGreat, I was going to also suggest something like minimum edit distance to compute distances between the input word and the true spelling of the word. But if you can imagine, for one input word, you would need to compute the minimum edit distance for over all words in the vocabulary, which could be computationally inefficient. If still want to pursue an unsupervised route, then I would then suggest trying to find conditions which can reduce the search space (i.e. number of words in vocabulary) when competing the distance so it's a bit more efficient. $\endgroup$
    – shepan6
    Jun 12, 2020 at 10:19
  • $\begingroup$ Hi @shepan6, can you suggest any tutorial which explains on how to do this? for now, computational efficiency doesn't matter $\endgroup$
    – The Great
    Jun 13, 2020 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.