I would like to give a context of what I did.
1) Let's say there are two dictionaries (dict A and dict B) each containing a list of words/terms as shown below.
2) Now my task is to find matching words for dict A in dict B
3) I use an automated tool (fuzzy-matching/similarity) to do the above task and the output looks like below
4) Once I get the output as above, you can see that there are some records with match % less than 100. It is totally possible that dict B didn't have the exact matching term. It's fine.
5) So, what I do is review terms that have match % less than 50. Meaning I take those terms (that are less that are 50% match) and check for a related term in dict B again. Doing this, I am able to update the output like below. Because we know through human experience that sore throat lozenge and strepsils are related (matching is better now when compared to earlier where it was mapped to orange (totally irrelevant)). So this problem is more of a semi-automated task rather than full-blown ML task
So, now my question is (not on NLP or ML but below)
1) But how can I prove that choosing 50% as the threshold for manual review is the right one? Because this is a subjective thing/ based on individual judgment. Meaning I could have chosen 30% or 40% as well, it could have saved my time in manually reviewing terms
2) Meaning, this 50% isn't written in stone but what I am looking for is some theory/mathematical/statistical approach through which I can arrive at this threshold value rather than based on my judgment/subjective which I cannot defend/justify?
Can you people share some views/techniques on how can this be done in a systematic approach?