# Next best predictions in decision tree

I am using decision tree classifier to predict some block selected based on the below data.

I am able to predict the "Block selected" column based on the data. How to get the second best, third best prediction and so on(I need a ordered list)? Can I get this using decision tree? Or should I be using a different model? Any thoughts on how to do this using python scikit-learn?

What you are looking for is a probabilistic classification for 2+ classes (multi class) i.e. evaluating a probability of being associated to a list of k classes given the observation. k = 4 blocks/classes for your example. Its a one vs many situation.

Binary classifiers (like logistic regression) have been extended to multi class problems.

One of the strategy used is to train/test your model k times, each time keeping one class in predicted block intact and turning other three predicted class to a single dummy class. Then use a binary probabilistic classifier(eg. logistic regression) to predict class probabilities. The order of class preference will be in decreasing order of class probabilities. For logistic regression and for other classifiers, this is inherently built in by scikit and applied by default:

lr = linear_model.LogisticRegression(multi_class = 'ovr')
# ovr is "one vs rest"
lr.predict_proba(test_features)


Yes, you can even use a pruned decision tree to get the class probabilities. But most probably you will not be able to get 2nd, 3rd... best predictions for most of your observations from a single tree due to the underlying splitting mechanism of algorithm.

 dt = tree.DecisionTreeClassifier(min_samples_split=25)
dt.predict_proba(test_features)


Therefore, rather then relying on a single decision three, better option would be to use a randomforest classifier to obtain proportion of votes going to each class while making a prediction for an observation.

In short, any multi-class classifier (or ensemble) which spits out a likelihood over multiple classes will do. Another eg. XGBoost as pointed out in below comment by bradS.

• Agree that you need multi-class classification. An alternative to random forest could be XGBoost, where one of the objective targets is multi:softprob - this will output the likelihood of each class, so you can choose first, second, third,..., best. – bradS May 11 '18 at 9:02
• @Mankind_008 I'm a beginner, could you please explain in more detail using my data as example how to train model 4 times using logistic regression. I couldn't understand what you mean by keeping predicted block intact and turning rest into dummy class. – Vamsiga May 11 '18 at 23:20
• @Vamsiga If you are starting learning you don't need to do it yourself, Scikit has built this functionality for many of the the classifiers. I modified my answer for more details regarding implementation. Also, what i meant was to turn your original labels/ multi-class column into binary classification problem. i.e. if your original label column was c_multi =[1,0,2,1,1] which has 3 classes into binary, c_bin = [d,0,d,d,d] where d is dummy class and its not equal to 0. – Mankind_008 May 12 '18 at 23:37

Remove the best block and run again in batches, removing the best selected at each round and placing is as the n best prediction.

• That should work I think. But If I remove the best block column should I also be removing the rows in which this block was selected? – Vamsiga May 10 '18 at 19:21
• My suggestion would be to leave the row. I cannot see a disadvantage. – StevenTheDataGuy May 10 '18 at 19:27
• Ok so.. let's say that after training the model it predicted that block 2 is the best suited for some data. Now if I remove the block 2 column and train the model again wouldn't the model predict it again as block 2? what am i missing here? – Vamsiga May 11 '18 at 23:14