3
$\begingroup$

I had asked a question regarding predictive analysis for marketing earlier. Prediction model for marketing to prospective customers (using pandas) Still have some doubts about it, but I have a doubt regarding the decision tree that I generated for the marketing data.

My aim is to predict if a lead will be won or lost, depending on how they were made aware of the product etc. I have a bool variable "Won", 0-sale had failed, 1-sale was made. Using a decision tree, I was able to generate a model, however, there are no leaves for cases that lead to Not Won.

Is this normal? Ive seen examples of the iris data set where all 3 features were represented in the tree, adn, hence am wondering if the approach Ive taken is correct.

In the dataset of 38000, there are roughly 1700 who have Won=1.

enter image description here

Im using pandas and my parameters for DecisionTreeClassifier are:

min_samples_split=2 min_samples_leaf=1 max_depth=3

I got then using grid search for parameter optimisation. I got a mean val of 0.95. If I use a bigger depth, the tree becomes too big to analyse.

Thanks

EDIT

Since Mark said to post some code, I am doing so

from sklearn.tree import DecisionTreeClassifier,export_graphviz
dt=DecisionTreeClassifier(min_samples_split=2,min_samples_leaf=1,max_depth=3)
dt.fit(x,y)
features=x.columns
#print(features)
with open("dt.dot", 'w') as f:
    export_graphviz(dt, out_file=f,feature_names=features,class_names=["Won","Lost"])
command = ["dot", "-Tpng", "dt.dot", "-o", "hello.png"]
try:
    subprocess.check_call(command)
except:
    exit("Could not run dot, ie graphviz, to produce visualization")

from sklearn.cross_validation import cross_val_score
scores=cross_val_score(dt,x,y,cv=10)
print(scores.mean())

This is the main training code, all previous lines are just munging

The cross validation score come to 0.95

enter image description here

Here is the csv snapshot

I use all values from "Won" onwards and am training for ""Won"

There was a single column X, which had many categorical values (20), "Won" being one of them.

Known' 'Recycled' 'Engaged' 'Prospect/MQL' 'Intern Transfer' 'MQL'
 'Working' 'Opportunity' nan 'Current Customer' 'Vendor' 'Disqualified'
 'Converted' 'SAL' 'SQL' 'engaged' 'working' 'Won' 'Web Registration'

'Inactive

However, they were all uniformly distributed ie. from 37000, all had almost the same number of observations. I used get_dummies to transform to numerical values, and dropped all the columns except "Won"

All the rest of the values value for things like designation, opp(money value scaled from 1-3) and other categories, which are all boolean

$\endgroup$
  • 1
    $\begingroup$ Any chance you can post a small chunk of the code you are using and a snapshot of your data? In my experience with trees and class imbalance, most of your terminal nodes should be "Not Won". Have you tried any other modeling techniques to see if they always predict "Won"? It may give some insight into whats going on in your code/data. $\endgroup$ – TBSRounder Apr 28 '16 at 12:39
  • 1
    $\begingroup$ hmm, thanks for updating. I'm not super familiar with pandas, so I'm hoping someone else might be able to find something, but the cross validation score of 0.95 ~(38000-1700)/38000 accuracy would support that the classifier is calling almost everything as "Not Won", which would mean your visualization is off. Does the order of class_names=["Won","Lost"] in your export_graphviz statement make a difference? I would think that the first position would correspond to 0, which should be "Lost", so it should be class_names=["Lost","Won"]? $\endgroup$ – TBSRounder Apr 28 '16 at 13:21
  • $\begingroup$ Well if thats the case, shucks!!! I understood the CV thing from this site Decision trees in python Im really not sure if the class thing is wrong, hope that someone who's good at pandas can help. Thanks a lot anyways!!! $\endgroup$ – TdBm Apr 28 '16 at 13:32
  • 1
    $\begingroup$ Consider looking into the class weights argument in the DecisionTreeClassifier. The issue is that the model will be most accurate by always predicting "Not Won" since that is the vast majority. By changing the class weights to weight "Won" observations more, you tell the model that misclassifying a "Won" observation is more important than just focusing on "Not Won". The weights you use will affect the model, so try some different ones and see when you start getting a better split of final predictions. $\endgroup$ – TBSRounder Apr 28 '16 at 13:46
1
$\begingroup$

You are predicting binary class: 0 - sale failed, 1 - sale succeed. In the decision tree, the information value is sorted. So for example in the first node, you have 35011 predictions of 0 and 1785 predictions of class 1.

But then, in your code you have this: class_names=["Won","Lost"]). So you are telling your decision tree that name of class 0 is "Won" and name of class 1 is "Lost". I assume it should be inverted.

So in fact, the model always predict 0 - sale failed. Which seems reasonable as you said that out of 38000 samples, only 1700 are class 1.

Further, your dataset is highly unbalanced. So when cross validation gives accuracy of 95%, it says nothing, because only if your model gives all prediction 0, you will have accuracy of...95%. For those cases, use models which can put weights to classes (SVM, Logistic regression or Random Forest classifier in scikit have this possibility). Think about undersample your dataset. Or add more of class 1 - dig in your database or do it artificially (SMOTE),

Plot confusion matrix and give CV different score measure - precision or recall, depends on your situation.

Last thing, don't expect that your decision tree with depth of 3 will perform well. It is nice to show to business, but it gives poor prediction.

$\endgroup$
  • $\begingroup$ Thanks HonzaB!!! I will now try to downsample the 'sale failed'. Is a 50-50 split OK? The biggest problem I have is even if I use another kind of prediction model, I don't know how to implement data visualisation for clustering, rand. forests etc. Any idea? $\endgroup$ – TdBm May 2 '16 at 12:37
  • $\begingroup$ Just try different settings for the undersampling. Take all samples from the undersampled class and add 2, 3, 4 times much of second class and see what works best. BUT! You must test on your original dataset and not on undersampled one. You cant really visualize black-box models with a lot of dimensions. Aim for confusion matrix or ROC curve. $\endgroup$ – HonzaB May 2 '16 at 13:43
  • $\begingroup$ I'm sorry, but could you please explain what it means to test on the org. dataset. For eg. I have 9/10 'lost's and 1/10 'won's. Hence I randomly sample an equal 1/10 from 'lost's and add it to the 'won's and train on this set. I don't understand what it means to try different settings of undersampling. Sorry for asking an obvious question.(Unobvious for me :-( ) PS doing the latter,I got a CV mean of 0.3, with std of 0.03. Is that good? I switched the lost won thing too. Thanks $\endgroup$ – TdBm May 2 '16 at 13:53
  • $\begingroup$ Dont worry, I have been there :). So train set: all your 'won' samples plus arbitraty number of 'lost' samples (this is what I mean by different settings of undersampling - how much 'lost' you will take to train set). And then test set - lets say 20% from original data with original proportion. I can't judge the metrics. Try a lot of models and parameters, different approaches. Maybe thats all you can get. Maybe you can get more. $\endgroup$ – HonzaB May 2 '16 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.