- I have a set of items.
- Two judges will subjectively rate each item giving a grade from 0 to 10.
- I have to select the best 10 items.
How do I rank the items so that the judges are equally important?
Suppose one of them is really rigorous, and grade the items with a lower value, with a median of 5 and a standard deviation of 1.
The other median is 8, but his grades are more spread, with a standard deviation of 2.
Here are examples of the grades distributions:
The second judge has more spread grades, so if I just calculate the mean and rank, a lower grade of Judge Two would eliminate one item. Judge One grades would just order the best items of Judge Two. A favorite of Judge One would hardly get into the favorite rank.
Just to rank the items of each judge would make it more fair, but wouldn't take the grade difference between the items.