# Text Mining - Similarity among words to determine thresholds

While trying to find how one place (Military High School) like a school is similar to another one (Military H School or Military High S), I used the LV algorithm to find the string distance and then converted them into a percentage on a calibrated scale.

stringdist(data[i,1],data[i,2],method = "lv")

Though this is admissible, I still need a more precise way to determine the threshold. The goal is to set a threshold of acceptance for a variation of a name from the standardized name, so that say if the threshold of this word is less than say 85%, it would be rejected. This exercise is part of data pre-processing.

Any suggestions.

Have you tried word2vec models ?. One of the main application is computing similar words, as they are very closer in the feature vector space. Thus apart from High_school and H_school, also secondary_school, public_school may also be obtained as closer words. The important thing is that a huge amount of sentences are needed for this unsupervised learning model.
Next, compute string distance as before on this data. Create a vector of threshold values, e.g., thresholds <- seq(from = 0, to = 20, by = 1), then for each threshold compute your performance metrics on your labeled data set (e.g., accuracy, precision, recall). Choose the threshold that maximizes your desired metric.