1
$\begingroup$

My data is a list of sentences, where each sentence contains between 1 and 4 words. These sentences are typed in manually so some of them contain typos and some additional words such as GmbH, GER etc.

However, I do know the set of valid sentences. As an example we assume this valid set is given by {Hello human, Horse, Hello bird} and the data (where some sentences contain typos and extra words) is given by

Hello human
Horse
Hello human GmbH
Hello human GmbH, GER
Horse GmbH
Horse
Hello humn
Hell humn
Hello human
Hello bird

I would like to give each sentence above an ID 1, 2 or 3 where 1 is for Hello world, 2 is Horse and 3 is Hello bird. But due to the typos and extra words such as GmbH, GER I cannot make a simple comparison between strings.

Is there a numerical technique within NLP or a related field that I can use to achieve this task?

$\endgroup$
2
$\begingroup$

If I understand correctly, you're looking for string similarity. There are several techniques available, the most simple is "edit distance" (aka levenshtein distance), which is the count of the minimum insertion/deletion/substitution/transposition operations needed to get from one string to the other.

For your particular task, I suspect "jaro-winkler similarity" would be better. JW is similar to ED, but was specifically designed for "entity resolution" (i.e. "record linkage"), which it looks like is what you're trying to accomplish. You can see a short demonstration of how this would work here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.