1
$\begingroup$

I am working on a sentiment analysis project involving tweets. I used a Kaggle dataset to train my model for sentiment analysis and want to use that trained model to predict the sentiment on an entirely separate group of tweets that are not included in the dataset.

The code is as follows (aside from some cleaning steps):

tweets = pd.read_csv("tweets.csv")
sentiment = pd.read_csv("sentimenttraining.csv")

#counter
def sentiment_counting(input):
    counter = CountVectorizer(strip_accents='unicode', stop_words=['and','the', 'but'])
    counts = counter.fit_transform(input)
    return counts

counts = sentiment_counting(sentiment["text"])

text_train, text_test, sentiment_train, sentiment_test = train_test_split(counts, sentiment["sentiment"], random_state = 100)

#sentiment
sentiment_model = MultinomialNB(alpha = 0.9)
sentiment_model.fit(text_train, sentiment_train)

score = sentiment_model.score(text_test, sentiment_test)
print(score)

tweet_counts = sentiment_counting(tweets["text"])

predictions = sentiment_model.predict(tweet_counts)

However, in the final line (asking for predictions) I get an error--"dimension mismatch"--because the dimension is entirely different. Of course the dimensions are different--this is a different set of tweets. How can I fix this problem?

$\endgroup$

1 Answer 1

0
$\begingroup$

In response to:

However, in the final line (asking for predictions) I get an error--"dimension mismatch"--because the dimension is entirely different. Of course the dimensions are different--this is a different set of tweets. How can I fix this problem?

While of course this is a different set of tweets, what that error message is really saying is that the number of features in your new tweet data set does not match the number of features in your training tweet set. Assuming you are using scikit-learn, as you can see from the documentation for the parameters for predict(X):

X: array-like of shape (n_samples, n_features)

Even though you are now analyzing different tweets, n_features, must remain the same because your original model was training using n_features, so how can the model do inference on your new tweets if they have a mismatched number of features? The model only knows how to give output values for input values with n_features because that is what it was trained to do!

If anything I am saying is confusing or you want more detail now how Naive Bayes Models work see here.

Hope this helps.

Documentation for sklearn.naive_bayes.MultinomialNB:

https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.