Our product has an onboarding questionnaire which asks the same 58 questions (with numeric answers) to every new user. That’s a lot of questions, so we’d love to reduce the number of questions we ask each new user.

We figure that instead of asking all of the questions every time, we could create a system that asks the most “informative” question at each point, given all of the user’s previous responses so far - i.e. the question that does the most to improve the accuracy of our prediction about how they would answer the remaining questions.

I visualise this as a multiway tree, where each node represents a question and each branch represents a range of answers to that question. Every user sees the question at the root node, then each subsequent answer they give defines how they traverse the tree.

The questions finish when they reach a leaf node (e.g. at depth 7), but even if they quit the onboarding process early, the questions they have been asked should still provide the best-possible prediction of how they would answer all of the remaining questions from the 58.

The question is, how do we construct this tree? We have data from 348 customers who answered all 58 questions, so it should be possible - but what is the best algorithm?

We tried the RandomForestRegressor from SciKit, but that doesn’t seem at all suitable for this problem, as the trees are not at all in this format. Like other algorithms we’ve looked at, it’s great for predicting based on their previous answers - but not for knowing which questions to ask.

Can anybody suggest an algorithm (or class of algorithms) that is able to construct this tree?

  • $\begingroup$ Hm, this gets a little trickier if you don't know how many questions they'll answer in total. You might find that there's one question that would let you predict the other 57 reasonably well, but that a different two questions would let you predict the other 56 even better. In this case, you'd ask different questions if you know they're only going to answer 1 or 2. I guess you'd need to take a greedy approach to select the next question sequentially, since you have to assume that might be the last question they answer. $\endgroup$ Jul 15 '21 at 14:19

What are the outcomes of this questionnaire? Is it a binary solution, or a set of multiple mutually exclusive classes (multi label) or a set of multiple overlapping classes (multi label & multi class)?

Assuming a binary or a multi label, decision trees with good visualization should give you a smaller trees. These may not make semantic sense, but if you know the subject, you should be able to create a new questionnaire that makes sense.

I use sklearn decision trees https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

the native visualization is pretty useful, for prettier and more detailed graphs I have used https://explained.ai/decision-tree-viz/

  • $\begingroup$ Thanks aflip. I'm not sure I completely understand your questions but I can give a bit more detail about the questionnaire. For each of the 58 statements, the customers give scores for Importance and Satisfaction (on a 1-5 scale), which are used to calculate a numeric "pain" score. We don't combine the statements or give them any 'response', we keep all the pain scores separately to generate recommendations later. Of course it's fine if an algorithm uses thresholds to divide the pain scores into classes; we only need to know the best question to ask next. $\endgroup$ Jun 11 '21 at 11:33
  • $\begingroup$ if you convert them into categories, what you'd get are trees that fit a particular category. pain rating 0-2 will have a different solution from pain rating 3-5, say. so while the decision tree that you get itself might not be implementable, you might be able to figure out the most important questions overall and the most useless ones $\endgroup$
    – aflip
    Jun 12 '21 at 6:12
  • $\begingroup$ Yes, I agree. The question is: what algorithm can build that tree? If the DecisionTreeClassifier can help, maybe we don't understand the appropriate way to use it. $\endgroup$ Jun 13 '21 at 8:44
  • $\begingroup$ 1. treat questions as features, 2. run classifier and look into feature importance to see which features are most important to which label 3. based on feature importance make a new tree (manually, maybe) $\endgroup$
    – aflip
    Jun 14 '21 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.