Text Mining - Similarity among words to determine thresholds

Question

While trying to find how one place (Military High School) like a school is similar to another one (Military H School or Military High S), I used the LV algorithm to find the string distance and then converted them into a percentage on a calibrated scale.

stringdist(data[i,1],data[i,2],method = "lv")

Though this is admissible, I still need a more precise way to determine the threshold. The goal is to set a threshold of acceptance for a variation of a name from the standardized name, so that say if the threshold of this word is less than say 85%, it would be rejected. This exercise is part of data pre-processing.

Any suggestions.

Answer 1

Have you tried word2vec models ?. One of the main application is computing similar words, as they are very closer in the feature vector space. Thus apart from High_school and H_school, also secondary_school, public_school may also be obtained as closer words. The important thing is that a huge amount of sentences are needed for this unsupervised learning model.

Ref: gensim, word2vec in java

Answer 2

In order to optimize the string distance threshold for maximizing classification accuracy in your data, if you don't already have a gold standard dataset, you will want to label some of your data. Depending on your situation, this could be as easy as you yourself hand-labeling string pairs as same or notsame. If you need more robust data, you would obviously want to use several labelers and use some sort of procedure to ensure high inter-coder reliability (I discuss this at length under data quality in this answer).

Next, compute string distance as before on this data. Create a vector of threshold values, e.g., thresholds <- seq(from = 0, to = 20, by = 1), then for each threshold compute your performance metrics on your labeled data set (e.g., accuracy, precision, recall). Choose the threshold that maximizes your desired metric.