Individual differences in response time (RT) experiment - searching for the right test

Question

Given a distribution of response times to different categorical variables, what's the best way to test for individual differences?

or more specifically:

There are 100 people pressing buttons in 10 colors, with more than 200k presses collected altogether. I want to test for individual differences in response-time (RT) to the buttons, what's the best way to do this?

(remember there are differences in response time in-between the different colors).

So far I thought of:

1. ANOVA - calculate a general (weighted) mean RT and compare to individual people.

2. The distribution level: compare the mean distribution of RT to individual distributions using:

   • chi-squared for goodness of fit
   • Kullback–Leibler divergence test
   • Kolmogorov–Smirnov test

3. Find a cool bayesian way to do it (which?)

4. Use a mixed-effect regression model, and calculate the marginal contribution of the people (R square test).

please help me decide - what's the best practice here? I'd love to learn about another way I didn't think of that would do the trick.

Answer 1

In most individual differences research, subjects are additionally coded along some dimension, such as young/old, low/high working memory, normal/cognitively impaired. Without this, you’re simply looking at individual variation, which doesn’t seem to address any question (what is your research question?). If you simply want to look at variation in overall RT on a subject level, I’d construct CDFs of mean RT collapsing over color, or perhaps for each color separately. This will simply demonstrate that people differ, and you’ll find the distributions adhere to roughly ex-Gaussian. If you mean to test differences between colors, you can use ANOVA or compare CDFs using K-S tests, but again there isn’t a clear research question here.