0
$\begingroup$

I used CountVectorizer and TfidfVectorizer seperately to vectorize text which is 100K reviews and passed the vector data to a Decision tree Classifier. Upon using _feature_importances__ attribute of Decision tree Classifier, the feature importance values for all my feature are just 0.0. But with the same dataset, I'm able to find feature importances for naives bayes and logisitic regression by using feature_prob for naive bayes and coef_ attributes for logisitic regression.

Other things I tried: 1. I tried changing ngram_range in countvectorizer 2. I tried limiting/not limiting min_df and max_feature parameters passed in countvectorizers But couldn't make it work.

Any help is appreciated

Code:

positivereviews = df[df.Score == 1]
negativereviews = df[df.Score == 0]

countvect = CountVectorizer(stop_words='english')

positivebow = countvect.fit(positivereviews.CleanedText[0:100000])

pos_xtrain = positivebow.transform(positivereviews.CleanedText[100000:200000])

pos_y = positivereviews.Score[100000:200000]

clf = DecisionTreeClassifier(max_depth=3, min_samples_split=2)

clf.fit(pos_xtrain, pos_y)

def show_most_informative_features(vectorizer, clf, n=20):
    feature_names = vectorizer.get_feature_names()
    coefs_with_fns = sorted(zip(clf.feature_importances_, feature_names))
    top = coefs_with_fns[:-(n + 1):-1]
    for (coef_1, fn_1)in top:
        print ("\t%.4f\t%-15s" % (coef_1, fn_1))


Counter(clf.feature_importances_)
Counter({0.0: 63514})

1) positivereviews is the dataframe which has features CleanedText and Score where CleanedText is text which is preprocessed and Score is either 1 in this set since I splitted positive and negative review using Score

2) I also searched online for this problem, but couldn't find any instance of this issue

EDIT1: can it be because since we are dealing with categorical feature in this problem, I might be overfitting badly and hence I do not see any value for any features?

Thanks in advance!!

$\endgroup$
0
$\begingroup$

I'm pretty sure that your feature importances are 0 because your classifier isn't doing any classifying. From the code, it looks like you're training only on positive examples, and giving the fit function a label vector that consists entirely of 1s. The classifier has no information; the decision rule is just "when given an example, predict 1".

There's no way to measure which features are most strongly associated with the label because they're all equally associated - there's only one label, so there's no way to associate the features with anything else.

Is there a reason you're not using the negative examples? It seems like you have the dataframe available.

When you ran naive bayes and logistic regression, did you also give those models only the positive examples?

$\endgroup$
  • $\begingroup$ Thanks! after your suggestion. Now I fitted the classifier with total x_train data instead of only fitting it with pos_xtrain which again is a 100K dataset which has roughly 87k positive reviews and 13k negative reviews, but classifier managed to find only 4 text features are non zeros. I basically time sorted the dataset and pulled out the first 100K reviews, vectorized them and performed column standardisation before fitting them. $\endgroup$ – karthikeyan mg Jan 4 '19 at 15:55
  • $\begingroup$ Q1) "Is there a reason you're not using the negative examples? It seems like you have the dataframe available" - Yes, To find the important feature in my positive reviews(which should give me positive words more often than cuss words) and how they differ in negative reviews Q2) "When you ran naive bayes and logistic regression, did you also give those models only the positive examples?" - Yes I did the same when I ran all my other models $\endgroup$ – karthikeyan mg Jan 4 '19 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.