I have built binary text classifier using SVM on TF-IDF for news articles(Sports: Non-Sports).

But I not sure how to classify new document using this model. Since TF-IDF is calculated based on the occurrence of a word in all other documents.

Do I have merge test and train data every time I receive a new document for classification? It will change the model as well every time.

Am I missing something? I think, although SVM on TF-IDF giving good results it can not be used in production.

Is there any other way to tackle this issue?

Lets take an example

Training Set:
Doc_1: Chelsea won the match. {Sports}
Doc_2: India won the third test match against Austrailia {Sports}
Doc_3: I want to sleep {Non-Sport}
Doc_4: 13 palace to see in Auckland {Non-Sport}

New Testing Set:
Doc_5: Climate change impacts in Austrailia

Now how can I find IDF score of "Austrailia" in Doc_5 without merging this document with training set?

Since Doc_5 contains the word "Austrailia", it will change the IDF score of "Australia" in Doc_1 will also change, thus model needs retraining


What is your model built in?

Most popular libraries have a score function separate from the training part. You should be able to just pass the new document to the score function of the trained model and get back the predicted class.

  • $\begingroup$ I am using sklearn feature_extraction, svn. I will look at score function here. Thanks :) $\endgroup$ – Eudie Mar 31 '17 at 13:49
  • $\begingroup$ Check out this example on the scikit-learn website: scikit-learn.org/stable/tutorial/basic/… See the line like "clf2.predict(" ? They've loaded the trained model and are passing it values to predict. The returned value is the prediction. The input should match the format of the trained set. As another poster wrote, any words that aren't found in the original TF-IDI should be ignored. $\endgroup$ – CalZ Mar 31 '17 at 14:53

So your TF-IDF is trained only using the training set. It will determine the frequency of the occurrence of words. If you show the TF-IDF a new word it has not yet seen then it will simply ignore it. It will only use words that are in its training set. So, NO you do not retrain your model after you have built it. Once, you go through the training stage for your TF-IDf that is the library of words that your algorithm can detect.

  • $\begingroup$ I am not worried new word but the IDF of the word in training set might change. I have added an example in the question. Please have a look $\endgroup$ – Eudie Mar 31 '17 at 13:52
  • $\begingroup$ So imagine it like this, the training period takes all the words in the training set and builds a dictionary. The computer will not be able to understand any additional words. So in your example, things like climate change and impact will be ignored. With your dummy example, the testing example would most likely be classified as sport because it contains the word Australia and the computer associated that word as Sport due to your training set. $\endgroup$ – JahKnows Mar 31 '17 at 13:55

Through TF-IDF you have got the important words which you have used to train your SVM Model. So when you pass in the test data, only the words(features) that are selected through TF-IDF are important and will be used by the SVM Model to predict the label

Just make sure that the matrix used to train SVM is similar for both training and testing

  • $\begingroup$ So if I have to classify 100 new documents daily using the model. So do I have to retrain every time? $\endgroup$ – Eudie Mar 31 '17 at 13:55
  • $\begingroup$ Once the model is built which has good performance, you dont need to train it every day. $\endgroup$ – JamesBond101 Apr 10 '17 at 4:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.