0
$\begingroup$

I am using decision tree classifier to predict some block selected based on the below data.

enter image description here

I am able to predict the "Block selected" column based on the data. How to get the second best, third best prediction and so on(I need a ordered list)? Can I get this using decision tree? Or should I be using a different model? Any thoughts on how to do this using python scikit-learn?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

What you are looking for is a probabilistic classification for 2+ classes (multi class) i.e. evaluating a probability of being associated to a list of k classes given the observation. k = 4 blocks/classes for your example. Its a one vs many situation.

Binary classifiers (like logistic regression) have been extended to multi class problems.

One of the strategy used is to train/test your model k times, each time keeping one class in predicted block intact and turning other three predicted class to a single dummy class. Then use a binary probabilistic classifier(eg. logistic regression) to predict class probabilities. The order of class preference will be in decreasing order of class probabilities. For logistic regression and for other classifiers, this is inherently built in by scikit and applied by default:

lr = linear_model.LogisticRegression(multi_class = 'ovr')
# ovr is "one vs rest"
lr.predict_proba(test_features)

refer to Multi-class capability built in scikit

Yes, you can even use a pruned decision tree to get the class probabilities. But most probably you will not be able to get 2nd, 3rd... best predictions for most of your observations from a single tree due to the underlying splitting mechanism of algorithm.

 dt = tree.DecisionTreeClassifier(min_samples_split=25)
 dt.predict_proba(test_features)

Therefore, rather then relying on a single decision three, better option would be to use a randomforest classifier to obtain proportion of votes going to each class while making a prediction for an observation.

In short, any multi-class classifier (or ensemble) which spits out a likelihood over multiple classes will do. Another eg. XGBoost as pointed out in below comment by bradS.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Agree that you need multi-class classification. An alternative to random forest could be XGBoost, where one of the objective targets is multi:softprob - this will output the likelihood of each class, so you can choose first, second, third,..., best. $\endgroup$
    – bradS
    Commented May 11, 2018 at 9:02
  • $\begingroup$ @Mankind_008 I'm a beginner, could you please explain in more detail using my data as example how to train model 4 times using logistic regression. I couldn't understand what you mean by keeping predicted block intact and turning rest into dummy class. $\endgroup$
    – Vamsiga
    Commented May 11, 2018 at 23:20
  • $\begingroup$ @Vamsiga If you are starting learning you don't need to do it yourself, Scikit has built this functionality for many of the the classifiers. I modified my answer for more details regarding implementation. Also, what i meant was to turn your original labels/ multi-class column into binary classification problem. i.e. if your original label column was c_multi =[1,0,2,1,1] which has 3 classes into binary, c_bin = [d,0,d,d,d] where d is dummy class and its not equal to 0. $\endgroup$
    – Mankind_2000
    Commented May 12, 2018 at 23:37
1
$\begingroup$

Remove the best block and run again in batches, removing the best selected at each round and placing is as the n best prediction.

Improve this answer
$\endgroup$
3
  • $\begingroup$ That should work I think. But If I remove the best block column should I also be removing the rows in which this block was selected? $\endgroup$
    – Vamsiga
    Commented May 10, 2018 at 19:21
  • $\begingroup$ My suggestion would be to leave the row. I cannot see a disadvantage. $\endgroup$
    – TheDataScienceNinja
    Commented May 10, 2018 at 19:27
  • $\begingroup$ Ok so.. let's say that after training the model it predicted that block 2 is the best suited for some data. Now if I remove the block 2 column and train the model again wouldn't the model predict it again as block 2? what am i missing here? $\endgroup$
    – Vamsiga
    Commented May 11, 2018 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.