I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm. Here's the process:
Full sample data (labeled) <br> Training (labeled)<br> Test (labeled)<br> Unseen (non-labeled)<br>
This is my current workflow:
Fit IDF model (idf-1) on full Sample data<br> Apply(Transform) idf-1 on full sample data<br> Split data set into Training and Test data<br> Fit ML model on Training data<br> Apply(Transform) model on Test data<br> Apply(Transform) idf-1 on Unseen data<br> Apply(Transform) model on Unseen data<br>
I read somewhere that I should split my data into training and test BEFORE fitting IDF model; Fit IDF only on training data and then use the same transformer to transform training and test data.
Why would you do that? What exactly do IDF learn during the fitting process that it can reuse to transform any new dataset. Perhaps, idea is to keep same value for
DF|t, D| while use new
TF|t, D| ?
Also, how often I will Fit (not transform) IDF model against new unseen data? let say my model is ready for prediction. I made n prediction using same IDF and Classifier model. After that I want to retrain model as I have new data now. Should I also retrain IDF then?