Value of features is zero in Decision tree Classifier

Question

I used CountVectorizer and TfidfVectorizer seperately to vectorize text which is 100K reviews and passed the vector data to a Decision tree Classifier. Upon using _feature_importances__ attribute of Decision tree Classifier, the feature importance values for all my feature are just 0.0. But with the same dataset, I'm able to find feature importances for naives bayes and logisitic regression by using feature_prob for naive bayes and coef_ attributes for logisitic regression.

Other things I tried: 1. I tried changing ngram_range in countvectorizer 2. I tried limiting/not limiting min_df and max_feature parameters passed in countvectorizers But couldn't make it work.

Any help is appreciated

Code:

positivereviews = df[df.Score == 1]
negativereviews = df[df.Score == 0]

countvect = CountVectorizer(stop_words='english')

positivebow = countvect.fit(positivereviews.CleanedText[0:100000])

pos_xtrain = positivebow.transform(positivereviews.CleanedText[100000:200000])

pos_y = positivereviews.Score[100000:200000]

clf = DecisionTreeClassifier(max_depth=3, min_samples_split=2)

clf.fit(pos_xtrain, pos_y)

def show_most_informative_features(vectorizer, clf, n=20):
    feature_names = vectorizer.get_feature_names()
    coefs_with_fns = sorted(zip(clf.feature_importances_, feature_names))
    top = coefs_with_fns[:-(n + 1):-1]
    for (coef_1, fn_1)in top:
        print ("\t%.4f\t%-15s" % (coef_1, fn_1))


Counter(clf.feature_importances_)
Counter({0.0: 63514})

1) positivereviews is the dataframe which has features CleanedText and Score where CleanedText is text which is preprocessed and Score is either 1 in this set since I splitted positive and negative review using Score

2) I also searched online for this problem, but couldn't find any instance of this issue

EDIT1: can it be because since we are dealing with categorical feature in this problem, I might be overfitting badly and hence I do not see any value for any features?

Thanks in advance!!

Answer 1

I'm pretty sure that your feature importances are 0 because your classifier isn't doing any classifying. From the code, it looks like you're training only on positive examples, and giving the fit function a label vector that consists entirely of 1s. The classifier has no information; the decision rule is just "when given an example, predict 1".

There's no way to measure which features are most strongly associated with the label because they're all equally associated - there's only one label, so there's no way to associate the features with anything else.

Is there a reason you're not using the negative examples? It seems like you have the dataframe available.

When you ran naive bayes and logistic regression, did you also give those models only the positive examples?