1
$\begingroup$

Our product has an onboarding questionnaire which asks the same 58 questions (with numeric answers) to every new user. That’s a lot of questions, so we’d love to reduce the number of questions we ask each new user.

We figure that instead of asking all of the questions every time, we could create a system that asks the most “informative” question at each point, given all of the user’s previous responses so far - i.e. the question that does the most to improve the accuracy of our prediction about how they would answer the remaining questions.

I visualise this as a multiway tree, where each node represents a question and each branch represents a range of answers to that question. Every user sees the question at the root node, then each subsequent answer they give defines how they traverse the tree.

The questions finish when they reach a leaf node (e.g. at depth 7), but even if they quit the onboarding process early, the questions they have been asked should still provide the best-possible prediction of how they would answer all of the remaining questions from the 58.

The question is, how do we construct this tree? We have data from 348 customers who answered all 58 questions, so it should be possible - but what is the best algorithm?

We tried the RandomForestRegressor from SciKit, but that doesn’t seem at all suitable for this problem, as the trees are not at all in this format. Like other algorithms we’ve looked at, it’s great for predicting based on their previous answers - but not for knowing which questions to ask.

Can anybody suggest an algorithm (or class of algorithms) that is able to construct this tree?

$\endgroup$
1
  • $\begingroup$ Hm, this gets a little trickier if you don't know how many questions they'll answer in total. You might find that there's one question that would let you predict the other 57 reasonably well, but that a different two questions would let you predict the other 56 even better. In this case, you'd ask different questions if you know they're only going to answer 1 or 2. I guess you'd need to take a greedy approach to select the next question sequentially, since you have to assume that might be the last question they answer. $\endgroup$ Jul 15, 2021 at 14:19

3 Answers 3

0
$\begingroup$

What are the outcomes of this questionnaire? Is it a binary solution, or a set of multiple mutually exclusive classes (multi label) or a set of multiple overlapping classes (multi label & multi class)?

Assuming a binary or a multi label, decision trees with good visualization should give you a smaller trees. These may not make semantic sense, but if you know the subject, you should be able to create a new questionnaire that makes sense.

I use sklearn decision trees https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

the native visualization is pretty useful, for prettier and more detailed graphs I have used https://explained.ai/decision-tree-viz/

$\endgroup$
4
  • $\begingroup$ Thanks aflip. I'm not sure I completely understand your questions but I can give a bit more detail about the questionnaire. For each of the 58 statements, the customers give scores for Importance and Satisfaction (on a 1-5 scale), which are used to calculate a numeric "pain" score. We don't combine the statements or give them any 'response', we keep all the pain scores separately to generate recommendations later. Of course it's fine if an algorithm uses thresholds to divide the pain scores into classes; we only need to know the best question to ask next. $\endgroup$ Jun 11, 2021 at 11:33
  • $\begingroup$ if you convert them into categories, what you'd get are trees that fit a particular category. pain rating 0-2 will have a different solution from pain rating 3-5, say. so while the decision tree that you get itself might not be implementable, you might be able to figure out the most important questions overall and the most useless ones $\endgroup$
    – aflip
    Jun 12, 2021 at 6:12
  • $\begingroup$ Yes, I agree. The question is: what algorithm can build that tree? If the DecisionTreeClassifier can help, maybe we don't understand the appropriate way to use it. $\endgroup$ Jun 13, 2021 at 8:44
  • $\begingroup$ 1. treat questions as features, 2. run classifier and look into feature importance to see which features are most important to which label 3. based on feature importance make a new tree (manually, maybe) $\endgroup$
    – aflip
    Jun 14, 2021 at 11:36
0
$\begingroup$

I'm not at all sure this is the ideal solution, but it is an easy one: just use a regression tree (e.g. DecisionTreeRegressor from sklearn), training it with both the independent and dependent variables (X and y) being the questions' "pain scores". At each split point, the feature you split on will of course give excellent information about that target variable, but it will also inform about the other targets, which is what you suggest as the rough idea in your post. The split that gets chosen will be the one that gives the best information about all the questions' answers on average. This is, as Nuclear Hogie mentions in a comment, a greedily built solution.

This is complicated a bit by your pain score being calculated from two separate scores, but I think splitting on and predicting pain scores may still be fine? Maybe the bigger issue is that you may split on the same question more than once, at different thresholds of pain. I suppose what you really want is to make more than a binary split, one child for every possible question response; the Quinlan family of trees (as opposed to CART) could do that.

$\endgroup$
0
$\begingroup$

Here's a suggestion, assuming all responses are on the same scale (e.g. users may respond with values between 1 and 5 to all 58 questions).

Picking the first question to display

  1. Calculate the mean and variance of all answers of each question (denote the variance of question $i$ as $V_i$)
  2. Pick the question with highest variance to display first, that is $\arg \max_i V_i$

Assuming that user responds, which question should be displayed 2nd? According to the same line of though, it should be the question with next-highest variance, but that variance is no longer $V_i$ since by responding to the first question, the user provided information that reduces the variances of all remaining question. How do we go about calculating these posterior variances?

In order to avoid the exponential complexity of a brute-force conditional-probabilities model of 58 random variables, I suggest making the following simplifying assumption: Question responses follow a multivariate normal distribution. This means that the mean vector and variance matrix are all the parameters you need to move forward.

Picking the 2nd question and onward

  1. After the first question (the one with highest response variance) has been answered, the variances of all other questions is $V_{i|1} = V_i*(1-\rho_{i,1}^2)$. Which means that you can calculate the new variances directly from the variance matrix. Find the one with maximum variance ($\arg\max_i V_{i|1}$), and display this question second.

The next question to display should be that with highest posterior variance given the choice of 1st and 2nd questions.

The easiest way to find which one it is, is to run linear regressions with the questions chosen for 1st and 2nd as the explanatory variables. The SSE (Sum of squared errors) of these regression (you run one such regression for each of the remaining 56 questions) is the posterior variance given the 1st and 2nd.

Since the question order is deterministic (does not depend on the choice of response), it is feasible to continue running such regressions to determine the fourth, fifth, sixth questions to display, and so forth.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.