0
$\begingroup$

My problem:

  1. I have a set of items.
  2. Two judges will subjectively rate each item giving a grade from 0 to 10.
  3. I have to select the best 10 items.

How do I rank the items so that the judges are equally important?

Suppose one of them is really rigorous, and grade the items with a lower value, with a median of 5 and a standard deviation of 1.

The other median is 8, but his grades are more spread, with a standard deviation of 2.

Here are examples of the grades distributions:

Judge One

Judge Two

The second judge has more spread grades, so if I just calculate the mean and rank, a lower grade of Judge Two would eliminate one item. Judge One grades would just order the best items of Judge Two. A favorite of Judge One would hardly get into the favorite rank.

Just to rank the items of each judge would make it more fair, but wouldn't take the grade difference between the items.

$\endgroup$
1
$\begingroup$

Procedure-1:

Convert them into percentage and then take average of outcome of both.

so for example for person A, judge-1 rated 5 out of 7 and judge-2 rated 7 out of 10,

converting judge-1 score to percentage i.e., $5/7 *100$ = 71%(rounded 71.45)

converting judge-2 score to percentage i.e., $7/10 *100$ = 70%

Now take average of both i.e., $71+70 / 2$ = 70.5%, now you can convert it back on to a scale from $1-10$ : This person A's overall rating is 7(rounded 7.05 ) out of 10.

Procedure-2:

You can normalize the rating of both the judges by using this formula,

$$z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)}$$

where $x=(x_1,...,x_n)$ and $z_i$ is now your $i^{th}$ normalized data. As a proof of concept (although you did not ask for it)

Procedure-3:

You can convert the 0-7 rating to 0-10 and then take average of both.

Once you get the outcome then you can rank them accordingly.

Do let me know if you have any doubts.

$\endgroup$
  • $\begingroup$ It won't work. Both judges rate from 0 to 10. One is more rigorous and give, as a general rule, lower grades. I can't just remap the grades. $\endgroup$ – neves Nov 30 '17 at 17:34
  • $\begingroup$ I put some images to make it clearer $\endgroup$ – neves Nov 30 '17 at 19:57
  • $\begingroup$ So you mean to say that one gives rating leniently, where as the other doesn't. so you can give 60% weightage to judge-2(who gives genuine rating) and 40% weightage to the other. Then you can get better ranking $\endgroup$ – Toros91 Dec 1 '17 at 1:38
  • $\begingroup$ I think I explained better the problem. $\endgroup$ – neves Dec 1 '17 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.