4
$\begingroup$

I have a dataset of the following form:

System A Rating System B Rating
4.5 5
3 4
5 3
etc. etc.

I have 155 such data points gathered using a survey. Which statistical test should I use to show statistical significance if one system has significantly better ratings than the other?

Thanks.

Improve this question
$\endgroup$
9
  • $\begingroup$ I would use Average value as a the rating and standard deviation as the reliability of the rating. $\endgroup$
    – Ubikuity
    Commented Jun 8, 2021 at 19:15
  • $\begingroup$ Do you mean that the rows correspond to different items, and for each item you have a rating for system A and system B? Do you have a gold standard indicating what the ratings should be? If not I don't see how you can determine which system is better. $\endgroup$
    – Erwan
    Commented Jun 8, 2021 at 22:22
  • $\begingroup$ Each row corresponds to a user who was rated both system A and system B. $\endgroup$
    – Michael Pulis
    Commented Jun 9, 2021 at 8:31
  • $\begingroup$ But do you have a way to determine which system is "better"? Normally you need a gold standard (also called ground truth) for that. otherwise you just have two different predictions but you don't know if they are good or bad. $\endgroup$
    – Erwan
    Commented Jun 9, 2021 at 16:59
  • $\begingroup$ They are not predictions. They are user ratings from two different systems. Each row represents a user, and how they rated the recommendations from system A, and system B. I want to know which is the best statistical measure to establish statistical significance, such as paired-t-test or wilcoxon ranked. $\endgroup$
    – Michael Pulis
    Commented Jun 9, 2021 at 19:06

2 Answers 2

Reset to default
2
$\begingroup$

Mann Whitney U Test (Wilcoxon Rank Sum Test) shall enable us to compare the ratings on System A and system B. This test compares the shape of each population and tells whether the two samples differ. If the shape is same, the null hypothesis is accepted. If the shape of one of distribution is different from the another distribution, the two systems have a significant difference in observed ratings.

Improve this answer
$\endgroup$
0
2
$\begingroup$

You need to perform some tests to identify the appropriate statistical measure for comparing the two distributions accurately.

  1. For each group/system, run a normality test to make sure that you are not dealing with an ultra exotic distribution to which central limit theorem does not apply (very unlikely).

  2. Calculate the variance for each group, in order to see whether you can use tests that make the assumption of equal variances. To statistically test if the variances can be assumed to be equal, you can perform a Levene's test.

If variances are equal you can move ahead with independent unpaired t-test, otherwise you should trust Welch's test. You can perform these tests online as well, e.g. here https://www.graphpad.com/quickcalcs/ttest1.cfm

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.