1
$\begingroup$

I'm training a model to classify tweets right now. Most of the text classification examples I have seen convert the tweets into tf-idf document term matrices as input for the model. However, this model should be able to identify newly collected tweets without retraining. Does it make sense to use tf-idf in this context? What is the correct way to turn tweets into feature vectors in this task?

$\endgroup$
1
  • 1
    $\begingroup$ The effectiveness mainly depends on the corpus that you have taken for training. If it's large enough that it encompasses a large vocabulary then it would be good enough. TF_IDF is basically a kind of a matrix that basically tells you what is the likely combination of a set of words. If your training data is a good representation like in any ML method, then you would not require retraining. $\endgroup$ Mar 3 '20 at 6:38
1
$\begingroup$

The problem is not really "new text", since by definition any classification model for text is meant to be applied to some new text. The problem is out of vocabulary words (OOV): the model will not be able to represent words that it didn't see in the training data.

The most simple way (and probably the most standard way) to deal with OOV in the test data is to completely remove them before representing the text as features.

Naturally OOV words can be a serious problem, especially in data such as Twitter where vocabulary evolves fast. But this issue is not related to using TF-IDF or not: any model trained at a certain point in time can only take into account the vocabulary in the training data, it cannot guess how future words are going to behave with respect to the class. The only solution for that is to use some form of re-training, for instance semi-supervised learning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.