Mathematical way of identifying wrong suggestions or outliers

Question

I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person.

Eg. Input1 -Donald Trump
30/100 classifiers returns politician as the class
20/100 classifiers returns business man as the class
10/100 classifiers returns leader as the class
10/100 classifiers returns american as the class
10/100 classifiers returns republican as the class
10/100 classifiers returns sportsman as the class
3/100 classifiers returns priest as the class
3/100 classifiers returns doctor as the class
2/100 classifiers returns engineer as the class
1/100 classifiers returns indian as the class
1/100 classifiers returns sportsman as the class

In the above case i take 10 votes as a threshold, i can somewhat correctly define Donald Trump, though a definition of sportsman might be wrong. However 10 seems to be a decent threshold

Input2 -Christiano Ronaldo
20/100 classifiers returns sportsman as the class
20/100 classifiers returns foot ball player as the class
13/100 classifiers returns real madrid as the class
13/100 classifiers returns manchesterunited as the class
12/100 classifiers returns juventus as the class
12/100 classifiers returns european as the class
2/100 classifiers returns portugese as the class
2/100 classifiers returns cricketer as the class
2/100 classifiers returns american as the class
2/100 classifiers returns chinese as the class
2/100 classifiers returns korean as the class

In the above example, if i take 12 votes as the threshold, it correctly defines Christiano Ronaldo, though we might be missing portugese tag since its vote is only 2. However we are doing a good job here i guess.

My problem is, if i have an api that returns the votes and class of famous persons this way, what is the best mathematical aproach to dynamically find the best possible threshold value above which i can say that the definition is correct and below which you need to have a look if the classes are correct

Answer 1

what is the best mathematical aproach to dynamically find the best possible threshold value above which i can say that the definition is correct and below which you need to have a look if the classes are correct

First, in general it's clear that there's no way to be certain that an answer is correct or incorrect: for instance if for some reason all the classifiers return the same wrong answer, then the threshold condition is satisfied but the answer is wrong.

Now the only way to find an optimal threshold to decide whether an answer is acceptable is to use a set of labelled examples in order to evaluate the answers of the classifiers. Why? Because the correctness of the answers cannot be determined by any mathematical formula.

So the goal of the game is to assess to what extent this set of classifiers can be trusted to give a correct answer, and that depends on the threshold. So the threshold is a parameter of the predicting system, and this parameter can be estimated (tuned) based on some validation data. The basic method consists in trying all the possible values for the parameter, evaluate the performance in every case and pick the value which obtains the highest performance. Note that there are many possible evaluation measures for such a scenario: the measure should be chosen carefully depending on the goal of the system (e.g. is it better to have one false positive answer or false negative?).