1
$\begingroup$

Here's the scenario, There's a database with thousands of single-option questions for testing a specific skills, and a large number of users (either professional or amateur in this skill), each of which answer 10 random questions from the database.

The only thing that I can think of is to differentiate question difficulty level according to the correct rate of each question. But how could I take fully use of other informations like:

  1. the correct rate from the user's perspective and feedback to its own influence to the difficulty level of questions (if user A answers 9 out of 10 questions correct, then the incorrect one (question_10) is more likely to be harder, than an user B answers 1 out of 10 questions correct and question_10 is in the incorrect set)
  2. the answer time for each question by each user

Could anyone give me some ideas on this model, like where should I delve more into to make the difficulty level of question more accurate? Great thanks!

$\endgroup$
1

2 Answers 2

0
$\begingroup$

I suggest you look at item response theory and the Rasch model. They construct a model that attempts to identify both (a) the proficiency/knowledge/ability of the user and (b) the level of difficulty of the question.

$\endgroup$
0
$\begingroup$

As @D.W. says either an item response theory (IRT) model or the Rasch model may be a good choice. Note that the Rasch model is very similar to the IRT model, with the main difference being that the Rasch model constrains all items to be equally related to the underlying construct(s).

Your typical IRT or Rasch model, however, does not incorporate response time, and for this much more complicated models are needed.

See Bock & Gibbons (2021) for a good intro to IRT, and see Bürkner (2019) for info on IRT models that incorporate response time.

References

Bock, R. D., & Gibbons, R. D. (2021). Item response theory. John Wiley & Sons.

Bürkner, P. C. (2019). Bayesian item response modeling in R with brms and Stan. arXiv preprint arXiv:1905.09501.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.