1
$\begingroup$

I'm starting at Data Science and, to get something going, I just ran the code from Siraj Raval's Intro to Data Science video. He implements a simple Decision Tree Classifier but I couldn't help but notice that, given the same training set, the classifier doesn't always yield the same prediction (nor the same fit apparently); which I happen to find terribly weird, since, from what I've learned, a Decision Tree is supposed to be deterministic.

The only thing I can think of that could be causing the randomness would be that the branches are being chosen at random at some point because 2 options might be identically valued. I would say this could be corrected with a little bit more training data, but even if I add 5 more people, nothing changes. Does anybody have an explanation for what's going on?

Following is the code (in Python) from the video in a for loop to count how many predictions for male and female the Decision Tree has yielded.

from sklearn import tree
from sklearn.svm import SVC

n_male_pred_tree = 0
n_female_pred_tree = 0

n_male_pred_svm = 0
n_female_pred_svm = 0

for i in range (1,1000):
    # This loop tests the consistency of the CLF
    # The Decision Tree is not very consistent (It's 50-50)
    X = [[181,80,44],[177,70,43],[160,60,38],[154,54,37],
         [166,65,40],[190,90,47],[175,64,39],[177,70,40],
         [159,55,37],[171,75,42],[181,85,43]]

    Y = ['male','female','female','female',
         'male','male','male','female',
         'male','female','male']

    tree_clf = tree.DecisionTreeClassifier()
    svm_clf = SVC()

    tree_clf.fit(X,Y)
    svm_clf.fit(X,Y)

    tree_prediction = tree_clf.predict([[190,70,43]])
    svm_prediction = svm_clf.predict([[190,70,43]])

    if tree_prediction == 'male':
        n_male_pred_tree += 1
    else:
        n_female_pred_tree += 1

    if svm_prediction == 'male':
        n_male_pred_svm += 1
    else:
        n_female_pred_svm += 1

print(f"MALE pred Tree: {n_male_pred_tree}")
print(f"FEMALE pred for Tree: {n_female_pred_tree}")
print("\n")
print(f"MALE pred for SVM: {n_male_pred_svm}")
print(f"FEMALE pred for SVM: {n_female_pred_svm}")
$\endgroup$

1 Answer 1

3
$\begingroup$

From sklearn:

The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

If you manually set the random_state variable when you create your tree object, you'll find that it does become deterministic.

In simpler terms, the data you are feeding it is a little small, and there are several splits that have the same information gain, so the split that is chosen is subject to random factors.

$\endgroup$
1
  • $\begingroup$ Cool! Thanks, I'll have to read the documentation more carefully next time, I guess (although it's difficult with my knowledge right now). I've found this other post which seems to be very in line with my question too. Apparently, the Decision Tree tries to mimic a Random Forest by default, and, as j.a. gartner mentioned, you can change that by fixing the random_state. The classification differs completely depending on the value of random_state (0 or 1). $\endgroup$ Aug 9, 2018 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.