# Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties.

Let's say we have 2 coaches, namely A and B.

Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "Weak". Coach B will rate the same 12 players.

If Coach A rated all the players as "Strong", and Coach B rated 10 players as "Strong" and 2 players as "Weak".

I have tried putting the results into a contingency table form and ran Somers D test. However the Somers D shows 0% which does not truly representing the correlation of the ratings. What test could we carry out to find the correlation between the ratings of the coaches?

I don't think it's worth doing any kind of significance test for such a small sample.

# option 1: pure categorical values

You could just count the number of identical ratings between the two ratings and divide by the total number of players

# option 2: take the order into account: weak < moderate < strong

Define a similarity measure such as:

• sim(X, X) = 1
• sim(weak,moderate) = 0.5
• sim(moderate,strong) = 0.5
• sim(weak,strong) = 0

Then use the same process as option 1 except that for every player you add sim(ratingA, ratingB), and divide by the total. This will also give you a normalized score between 0 and 1 which is a bit more precise (the value will be higher or equal than in option 1).

You are describing the field of inter-rater reliability.

There are a variety of statistical tests for inter-rater reliability. One of the most common for that scenario is Cohen's kappa.

You could also try a chi-squared test for categorical data to see if there is a relationship, and then use Cramer's V to measure the strength of the association.