I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how can I represent a new document with the Document-Term Matrix since some terms might not be included in training data?
If you choose to use scikit-learn's CountVectorizer, words that appear in the test dataset but not in the training dataset are automatically ignored.
fit_transform method is called on training data creating the document-term matrix. The
transform only method is called on the test data which transforms those documents to document-term matrix created in training, automatically dropping any new terms.