# Spark MLLib - how to re-use TF-IDF model

I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm. Here's the process:

Datasets:

Full sample data (labeled) <br>
Training (labeled)<br>
Test (labeled)<br>
Unseen (non-labeled)<br>


This is my current workflow:

Fit IDF model (idf-1) on full Sample data<br>
Apply(Transform) idf-1 on full sample data<br>
Split data set into Training and Test data<br>
Fit ML model on Training data<br>
Apply(Transform) model on Test data<br>
Apply(Transform) idf-1 on Unseen data<br>
Apply(Transform) model on Unseen data<br>


I read somewhere that I should split my data into training and test BEFORE fitting IDF model; Fit IDF only on training data and then use the same transformer to transform training and test data.

Why would you do that? What exactly do IDF learn during the fitting process that it can reuse to transform any new dataset. Perhaps, idea is to keep same value for |D| and DF|t, D| while use new TF|t, D| ?

Also, how often I will Fit (not transform) IDF model against new unseen data? let say my model is ready for prediction. I made n prediction using same IDF and Classifier model. After that I want to retrain model as I have new data now. Should I also retrain IDF then?

tf-idf will learn a vocabulary, idf, and some will also learn stop words (based on min_df, max_df, max_features). Read over sklearn's TfidfVectorizer and you can see the attributes that the fit method will set.

When you expose a trained tf-idf to new data it will transform that data into a vector of the same size as your original data using the vocabulary to construct Term_Counts which are then converted into your tf vector. The value in this is that you can use another model to predict an outcome based on the tf_idf as each new document will have the same size tf_idf vector as the documents you used to train the model. Otherwise you couldn't use it to make a prediction! For example with a naive bayes classifying tfidf:

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(X_train)
nb = MultinomialNB()
nb.fit(X, y_train)

# When you receive a new document
X = tfidf.transform(new_doc)
prediction = nb.predict_proba(X)


And I don't think you would want to refit the model. If you want some kind of continuous real-time update consider implementing a bayesian update

• Thanks for the link to sklearn doc. Just to clarify in your example, I assume new_doc is a doc from your test set as you want to first run against test set to evaluate classifier. – nir Nov 3 '16 at 23:29
• It could be. If you think of this as a production system it would be a new observation. Say for example you are looking at email content as part of a spam filter. The new doc would be a new email that needs to be identified as spam or not. – Grr Nov 3 '16 at 23:34
• by the way, for n-fold Cross-validation you would train idf on training data for each n-folds right? – nir Nov 4 '16 at 15:53
• So the n-folds would all be from the original training set and for each cv the model would be fit to the train from the train/test split within that data. So if I understand you correctly yes. – Grr Nov 4 '16 at 17:08