2
$\begingroup$

I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case:

program_metrics = [0.1, 0.3, 0.2, 0.1, 0.5]

This list is ordered by case, i.e. A's accuracy is 0.1, B's accuracy is 0.2, etc.

I then take these images, and ask some experts to provide a metric between 0 and 1 to determine how accurate these five cases are. I make a list of the expert's metrics:

expert_metrics = [0.3, 0.5, 0.1, 0.5, 0.2]

This list is also ordered by case, i.e. the experts consider A's accuracy to be 0.3, B's accuracy to be 0.5, etc.

I would like to have a metric to measure how correlated these two sets of data are, taking into account the pairs of data, which metric could I use? Would the Pearson Correlation Coefficient be of use here?

I want to determine whether there is some sort of similarity between the program's metrics and the experts' metrics.

$\endgroup$
1
  • $\begingroup$ It would help if you said what you want to learn about your lists when you do a comparison. For instance, do you want to regard the experts as the “truth” (or some kind of “gold standard”) and check by how much the program deviates from that? Do you want to consider the program to be the gold standard and check if the experts agree with the program? Do you want to do something else? $\endgroup$
    – Dave
    Mar 14, 2023 at 10:52

2 Answers 2

0
$\begingroup$

The data is correlated in terms of paired data. Also,the number of pairs is small.Hence,t-test is recommended.

$\endgroup$
0
$\begingroup$

As I discuss here, Pearson correlation (Spearman, too) can miss differences. In the extreme, there could be a perfect Pearson correlation of $1$ despite the program and expert ratings disagreeing. Depending on your goals, you might find this acceptable, but it is important to be aware. You might find the below equation to be helpful, as it relates to squared correlation in simple circumstances but also detects deviations between the two lists.

$$ 1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

I am not sure which list should be regarded as the “truth” ($y$) and which should be considered the “prediction” ($\hat y$), but I would think that the experts make sense to regard as the “truth”.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.