1
$\begingroup$

I have built binary text classifier using SVM on TF-IDF for news articles(Sports: Non-Sports).

But I not sure how to classify new document using this model. Since TF-IDF is calculated based on the occurrence of a word in all other documents.

Do I have merge test and train data every time I receive a new document for classification? It will change the model as well every time.

Am I missing something? I think, although SVM on TF-IDF giving good results it can not be used in production.

Is there any other way to tackle this issue?

Lets take an example

Training Set:
Doc_1: Chelsea won the match. {Sports}
Doc_2: India won the third test match against Austrailia {Sports}
Doc_3: I want to sleep {Non-Sport}
Doc_4: 13 palace to see in Auckland {Non-Sport}

New Testing Set:
Doc_5: Climate change impacts in Austrailia

Now how can I find IDF score of "Austrailia" in Doc_5 without merging this document with training set?

Since Doc_5 contains the word "Austrailia", it will change the IDF score of "Australia" in Doc_1 will also change, thus model needs retraining

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

What is your model built in?

Most popular libraries have a score function separate from the training part. You should be able to just pass the new document to the score function of the trained model and get back the predicted class.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am using sklearn feature_extraction, svn. I will look at score function here. Thanks :) $\endgroup$
    – Eudie
    Commented Mar 31, 2017 at 13:49
  • $\begingroup$ Check out this example on the scikit-learn website: scikit-learn.org/stable/tutorial/basic/… See the line like "clf2.predict(" ? They've loaded the trained model and are passing it values to predict. The returned value is the prediction. The input should match the format of the trained set. As another poster wrote, any words that aren't found in the original TF-IDI should be ignored. $\endgroup$
    – CalZ
    Commented Mar 31, 2017 at 14:53
1
$\begingroup$

So your TF-IDF is trained only using the training set. It will determine the frequency of the occurrence of words. If you show the TF-IDF a new word it has not yet seen then it will simply ignore it. It will only use words that are in its training set. So, NO you do not retrain your model after you have built it. Once, you go through the training stage for your TF-IDf that is the library of words that your algorithm can detect.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am not worried new word but the IDF of the word in training set might change. I have added an example in the question. Please have a look $\endgroup$
    – Eudie
    Commented Mar 31, 2017 at 13:52
  • $\begingroup$ So imagine it like this, the training period takes all the words in the training set and builds a dictionary. The computer will not be able to understand any additional words. So in your example, things like climate change and impact will be ignored. With your dummy example, the testing example would most likely be classified as sport because it contains the word Australia and the computer associated that word as Sport due to your training set. $\endgroup$
    – JahKnows
    Commented Mar 31, 2017 at 13:55
0
$\begingroup$

Through TF-IDF you have got the important words which you have used to train your SVM Model. So when you pass in the test data, only the words(features) that are selected through TF-IDF are important and will be used by the SVM Model to predict the label

Just make sure that the matrix used to train SVM is similar for both training and testing

Improve this answer
$\endgroup$
2
  • $\begingroup$ So if I have to classify 100 new documents daily using the model. So do I have to retrain every time? $\endgroup$
    – Eudie
    Commented Mar 31, 2017 at 13:55
  • $\begingroup$ Once the model is built which has good performance, you dont need to train it every day. $\endgroup$
    – JamesBond101
    Commented Apr 10, 2017 at 4:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.