1
$\begingroup$

I need to find the correlation between the ratings given by two parties.

Let's say we have 2 coaches, namely A and B.

Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "Weak". Coach B will rate the same 12 players.

If Coach A rated all the players as "Strong", and Coach B rated 10 players as "Strong" and 2 players as "Weak".

I have tried putting the results into a contingency table form and ran Somers D test. However the Somers D shows 0% which does not truly representing the correlation of the ratings. What test could we carry out to find the correlation between the ratings of the coaches?

$\endgroup$
0
$\begingroup$

I don't think it's worth doing any kind of significance test for such a small sample.

option 1: pure categorical values

You could just count the number of identical ratings between the two ratings and divide by the total number of players

option 2: take the order into account: weak < moderate < strong

Define a similarity measure such as:

  • sim(X, X) = 1
  • sim(weak,moderate) = 0.5
  • sim(moderate,strong) = 0.5
  • sim(weak,strong) = 0

Then use the same process as option 1 except that for every player you add sim(ratingA, ratingB), and divide by the total. This will also give you a normalized score between 0 and 1 which is a bit more precise (the value will be higher or equal than in option 1).

| improve this answer | |
$\endgroup$
0
$\begingroup$

You are describing the field of inter-rater reliability.

There are a variety of statistical tests for inter-rater reliability. One of the most common for that scenario is Cohen's kappa.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.