0
$\begingroup$

I am working on a classifier for some twitter data to predict who it was tweeted by. I am only using the text of the tweets to build the model. After all text related preprocessing here is how I prepare and vectorise train and test data

x_train, x_test, y_train, y_test = train_test_split(tweets["tweet"], 
                                                    tweets["label"], test_size = 0.3, random_state = 42)

count_vect = CountVectorizer(stop_words='english')
transformer = TfidfTransformer(norm='l2',sublinear_tf=True)
x_train_counts = count_vect.fit_transform(x_train)
x_train_tfidf = transformer.fit_transform(x_train_counts)
x_test_counts = count_vect.transform(x_test)
x_test_tfidf = transformer.transform(x_test_counts)
print(x_train_counts.shape)

The output shape has 3792 features.

I want to be able to do something like

model.predict("enter tweet here")

And have the predicted label as an output.

However when I try to use a random tweet and vectorise it - the number of features is only 19, even if I put that tweet though all the same preprocessing steps, so when I try to run the prediction I get an error that number of features doesn't match.

I don't fully understand where exactly are either of these numbers coming from (and why the first one is so large) but also what would I have to do to make this work?

Thank you!

$\endgroup$

1 Answer 1

1
$\begingroup$

If you obtained only 19 features it's likely because you used fit_transform instead of transform for your instance. It's important to understand what the 'TfidfTransformer' does:

  • In the 'fitting' part, it assigns an index $i$ to every word in the training data vocabulary. 3792 is the size of your vocabulary. Note that it's common to restrict the size by using min_df in order to avoid overfitting.
  • In the 'transforming' part, every document (tweet) is represented as a vector over the full vocabulary, with every word in the document represented by the TFIDF weight at the corresponding index $i$.

In your code the test set is correctly transformed using the previously fitted transformer (i.e. with the training vocabulary). You should do the same for any new document as well, so that they are represented in the same way (there might be out of vocabulary words which are ignored).

Note that TDFIDF only provides a representation of the text. In order to train a model and apply it, you also need to use these TFIDF vectors with a classification algorithm, see for example sklearn documentation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.