I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm. Here's the process:


Full sample data (labeled) <br>
Training (labeled)<br>
Test (labeled)<br>
Unseen (non-labeled)<br>

This is my current workflow:

Fit IDF model (idf-1) on full Sample data<br>
Apply(Transform) idf-1 on full sample data<br>
Split data set into Training and Test data<br>
Fit ML model on Training data<br>
Apply(Transform) model on Test data<br>
Apply(Transform) idf-1 on Unseen data<br>
Apply(Transform) model on Unseen data<br>

I read somewhere that I should split my data into training and test BEFORE fitting IDF model; Fit IDF only on training data and then use the same transformer to transform training and test data.

Why would you do that? What exactly do IDF learn during the fitting process that it can reuse to transform any new dataset. Perhaps, idea is to keep same value for |D| and DF|t, D| while use new TF|t, D| ?

Also, how often I will Fit (not transform) IDF model against new unseen data? let say my model is ready for prediction. I made n prediction using same IDF and Classifier model. After that I want to retrain model as I have new data now. Should I also retrain IDF then?


1 Answer 1


tf-idf will learn a vocabulary, idf, and some will also learn stop words (based on min_df, max_df, max_features). Read over sklearn's TfidfVectorizer and you can see the attributes that the fit method will set.

When you expose a trained tf-idf to new data it will transform that data into a vector of the same size as your original data using the vocabulary to construct Term_Counts which are then converted into your tf vector. The value in this is that you can use another model to predict an outcome based on the tf_idf as each new document will have the same size tf_idf vector as the documents you used to train the model. Otherwise you couldn't use it to make a prediction! For example with a naive bayes classifying tfidf:

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(X_train)
nb = MultinomialNB()
nb.fit(X, y_train)

# When you receive a new document
X = tfidf.transform(new_doc)
prediction = nb.predict_proba(X)

And I don't think you would want to refit the model. If you want some kind of continuous real-time update consider implementing a bayesian update

  • $\begingroup$ Thanks for the link to sklearn doc. Just to clarify in your example, I assume new_doc is a doc from your test set as you want to first run against test set to evaluate classifier. $\endgroup$
    – nir
    Nov 3, 2016 at 23:29
  • $\begingroup$ It could be. If you think of this as a production system it would be a new observation. Say for example you are looking at email content as part of a spam filter. The new doc would be a new email that needs to be identified as spam or not. $\endgroup$
    – Grr
    Nov 3, 2016 at 23:34
  • $\begingroup$ by the way, for n-fold Cross-validation you would train idf on training data for each n-folds right? $\endgroup$
    – nir
    Nov 4, 2016 at 15:53
  • 1
    $\begingroup$ So the n-folds would all be from the original training set and for each cv the model would be fit to the train from the train/test split within that data. So if I understand you correctly yes. $\endgroup$
    – Grr
    Nov 4, 2016 at 17:08
  • $\begingroup$ is it possible to re-use the model in pyspark (not sklearn)? $\endgroup$
    – hoang tran
    Feb 2, 2021 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.