I would like to give a context of what I did.

1) Let's say there are two dictionaries (dict A and dict B) each containing a list of words/terms as shown below.

enter image description here

2) Now my task is to find matching words for dict A in dict B

3) I use an automated tool (fuzzy-matching/similarity) to do the above task and the output looks like below

enter image description here

4) Once I get the output as above, you can see that there are some records with match % less than 100. It is totally possible that dict B didn't have the exact matching term. It's fine.

5) So, what I do is review terms that have match % less than 50. Meaning I take those terms (that are less that are 50% match) and check for a related term in dict B again. Doing this, I am able to update the output like below. Because we know through human experience that sore throat lozenge and strepsils are related (matching is better now when compared to earlier where it was mapped to orange (totally irrelevant)). So this problem is more of a semi-automated task rather than full-blown ML task

enter image description here

So, now my question is (not on NLP or ML but below)

1) But how can I prove that choosing 50% as the threshold for manual review is the right one? Because this is a subjective thing/ based on individual judgment. Meaning I could have chosen 30% or 40% as well, it could have saved my time in manually reviewing terms

2) Meaning, this 50% isn't written in stone but what I am looking for is some theory/mathematical/statistical approach through which I can arrive at this threshold value rather than based on my judgment/subjective which I cannot defend/justify?

Can you people share some views/techniques on how can this be done in a systematic approach?

  • $\begingroup$ Unfortunately there really isn't because of the way your matches can be interpreted conceptually. In your case a 0% match was the best solution while even a 67% match (orange flavor) might have been completely irrelevant. Your algorithm looks only for "looks the same" but your evaluation metric is "means the same". Seeing as these two metrics can diverge quite quickly you might have to manually check everything that isn't an almost 100% match. $\endgroup$ – Fnguyen May 19 at 9:56
  • $\begingroup$ Yeah but when there is 70, 80% match, we can still be quite okay that it matches most of the words, so probably they mean the same (but not exactly matching). Now the context I gave above was an example. But can you share me the approaches how people usually replace human judgement with mathematical approaches or theories? Can help please $\endgroup$ – The Great May 19 at 9:59

When it comes to choosing a threshold, I can see 3 approaches:

  1. Make an educated guess

This is what you are currently doing. You pick a value and would need to argue why this is a reasonable threshold. Obviously, the argument is as strong as the assumptions you make.

  1. Unsupervised way

If you compute the matching score for all pairs between A and B, you can analyze the histogram of these scores. Most likely, you will have quite a lot of scores near 0 and a few scores near 100. Then, you can decide to pick a threshold. This itself can be done in different ways. You can pick the median/mean matching score, this becomes close to option 1 but at least, the number comes from a specific mathematical concept. You can use the Jenks-Fisher algorithm (here is a Python implementation). In summary, the algorithm will find a threshold that splits your data into clusters which minimize the intra-cluster deviation. This would be better than the median/mean since it's expected that the data will be quite skewed.

  1. Supervised way

If you somehow have access to which matches are confirmed/overturned by humans, you could use entropy/information gain to find the best threshold. This is similar to building a decision tree of depth 1 (a decision stump), where your input is the matching score and the target is a binary variable (whether or not a human says this is a match).

So your data would be something like this:

score, is_match
0.0, 0
0.1, 0
0.2, 1
0.3, 0
0.4, 1
0.5, 0
0.6, 0
0.7, 0
0.8, 1
0.9, 1
1.0, 1

My personal recommendation would be to use the Jenks-Fisher algorithm (option 2)

| improve this answer | |
  • $\begingroup$ Hi, Thanks for the response. Upvoted. Reading about point 2. But can you share info on how can point 3 be done? $\endgroup$ – The Great May 19 at 10:40
  • $\begingroup$ Meaning, I know about decision tree. It checks for a condition at each node and then splits them as Yes, no. In this case, how can it be used? Any simple example will really be helpful to me a lot. Yes, we do have some records which are confirmed as correct by human and more records confirmed as incorrect by human as well.. $\endgroup$ – The Great May 19 at 10:41
  • $\begingroup$ I added an example of what the data would look like. It essentially becomes a binary classification problem where your target is whether or not a human thinks it's a match, and your input feature is the matching score. You could technically train any ML model, but because it's a univariate problem, a decision tree would be the simplest statistically sound option. You could theoretically add more features, such as the length of both strings. I wouldn't be surprised that a high matching score has a higher chance to be a real match if both strings are long $\endgroup$ – Valentin Calomme May 19 at 10:47
  • 1
    $\begingroup$ Thanks a lot. Gonna try this and will mark the answer soon. Your help is very much appreciated $\endgroup$ – The Great May 19 at 10:55
  • 1
    $\begingroup$ Yes, you can! You can do prediction with a single feature. However, if you only have a single feature, a simple statistical approach is best. What I suggested (the decision stump) is just the "ML name" for a statistical way to find a split in univariate data $\endgroup$ – Valentin Calomme May 19 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.