Given a distribution of response times to different categorical variables, what's the best way to test for individual differences?

or more specifically:

There are 100 people pressing buttons in 10 colors, with more than 200k presses collected altogether. I want to test for individual differences in response-time (RT) to the buttons, what's the best way to do this?

(remember there are differences in response time in-between the different colors).

So far I thought of:

  1. ANOVA - calculate a general (weighted) mean RT and compare to individual people.
  2. The distribution level: compare the mean distribution of RT to individual distributions using:

    • chi-squared for goodness of fit
    • Kullback–Leibler divergence test
    • Kolmogorov–Smirnov test
  3. Find a cool bayesian way to do it (which?)

  4. Use a mixed-effect regression model, and calculate the marginal contribution of the people (R square test).

please help me decide - what's the best practice here? I'd love to learn about another way I didn't think of that would do the trick.

  • $\begingroup$ This question may find a larger audience on the stats stack exchange. stats.stackexchange.com $\endgroup$
    – kbrose
    May 21, 2018 at 23:10

1 Answer 1


In most individual differences research, subjects are additionally coded along some dimension, such as young/old, low/high working memory, normal/cognitively impaired. Without this, you’re simply looking at individual variation, which doesn’t seem to address any question (what is your research question?). If you simply want to look at variation in overall RT on a subject level, I’d construct CDFs of mean RT collapsing over color, or perhaps for each color separately. This will simply demonstrate that people differ, and you’ll find the distributions adhere to roughly ex-Gaussian. If you mean to test differences between colors, you can use ANOVA or compare CDFs using K-S tests, but again there isn’t a clear research question here.

  • $\begingroup$ Thanks for the answer, the research question asks whether individual variance exists in this specific task. The goal is to possibly develop a feature that addresses the differences in response time to specific buttons. Hence the search is for a tool that would allow to create a hierarchy of variance between the different buttons on one clear scale. $\endgroup$
    – Sharonio
    May 22, 2018 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.