How can we shorten our questionnaire to only ask the most informative question at each point?

Question

Our product has an onboarding questionnaire which asks the same 58 questions (with numeric answers) to every new user. That’s a lot of questions, so we’d love to reduce the number of questions we ask each new user.

We figure that instead of asking all of the questions every time, we could create a system that asks the most “informative” question at each point, given all of the user’s previous responses so far - i.e. the question that does the most to improve the accuracy of our prediction about how they would answer the remaining questions.

I visualise this as a multiway tree, where each node represents a question and each branch represents a range of answers to that question. Every user sees the question at the root node, then each subsequent answer they give defines how they traverse the tree.

The questions finish when they reach a leaf node (e.g. at depth 7), but even if they quit the onboarding process early, the questions they have been asked should still provide the best-possible prediction of how they would answer all of the remaining questions from the 58.

The question is, how do we construct this tree? We have data from 348 customers who answered all 58 questions, so it should be possible - but what is the best algorithm?

We tried the RandomForestRegressor from SciKit, but that doesn’t seem at all suitable for this problem, as the trees are not at all in this format. Like other algorithms we’ve looked at, it’s great for predicting based on their previous answers - but not for knowing which questions to ask.

Can anybody suggest an algorithm (or class of algorithms) that is able to construct this tree?

Answer 1

What are the outcomes of this questionnaire? Is it a binary solution, or a set of multiple mutually exclusive classes (multi label) or a set of multiple overlapping classes (multi label & multi class)?

Assuming a binary or a multi label, decision trees with good visualization should give you a smaller trees. These may not make semantic sense, but if you know the subject, you should be able to create a new questionnaire that makes sense.

I use sklearn decision trees https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

the native visualization is pretty useful, for prettier and more detailed graphs I have used https://explained.ai/decision-tree-viz/

Answer 2

I'm not at all sure this is the ideal solution, but it is an easy one: just use a regression tree (e.g. DecisionTreeRegressor from sklearn), training it with both the independent and dependent variables (X and y) being the questions' "pain scores". At each split point, the feature you split on will of course give excellent information about that target variable, but it will also inform about the other targets, which is what you suggest as the rough idea in your post. The split that gets chosen will be the one that gives the best information about all the questions' answers on average. This is, as Nuclear Hogie mentions in a comment, a greedily built solution.

This is complicated a bit by your pain score being calculated from two separate scores, but I think splitting on and predicting pain scores may still be fine? Maybe the bigger issue is that you may split on the same question more than once, at different thresholds of pain. I suppose what you really want is to make more than a binary split, one child for every possible question response; the Quinlan family of trees (as opposed to CART) could do that.

Answer 3

Here's a suggestion, assuming all responses are on the same scale (e.g. users may respond with values between 1 and 5 to all 58 questions).

Picking the first question to display

1. Calculate the mean and variance of all answers of each question (denote the variance of question $i$ as $V_i$)
2. Pick the question with highest variance to display first, that is $\arg \max_i V_i$

Assuming that user responds, which question should be displayed 2nd? According to the same line of though, it should be the question with next-highest variance, but that variance is no longer $V_i$ since by responding to the first question, the user provided information that reduces the variances of all remaining question. How do we go about calculating these posterior variances?

In order to avoid the exponential complexity of a brute-force conditional-probabilities model of 58 random variables, I suggest making the following simplifying assumption: Question responses follow a multivariate normal distribution. This means that the mean vector and variance matrix are all the parameters you need to move forward.

Picking the 2nd question and onward

1. After the first question (the one with highest response variance) has been answered, the variances of all other questions is $V_{i|1} = V_i*(1-\rho_{i,1}^2)$. Which means that you can calculate the new variances directly from the variance matrix. Find the one with maximum variance ($\arg\max_i V_{i|1}$), and display this question second.

The next question to display should be that with highest posterior variance given the choice of 1st and 2nd questions.

The easiest way to find which one it is, is to run linear regressions with the questions chosen for 1st and 2nd as the explanatory variables. The SSE (Sum of squared errors) of these regression (you run one such regression for each of the remaining 56 questions) is the posterior variance given the 1st and 2nd.

Since the question order is deterministic (does not depend on the choice of response), it is feasible to continue running such regressions to determine the fourth, fifth, sixth questions to display, and so forth.