Avoid leakage in NLP extraction

Question

What is best practice for applying traditional NLP extraction techniques a pre-processing for ML models?

Given a pipeline:

1. Collect raw data.
2. Parse full data set with a variety of traditional NLP techniques, to create model-compatible features (e.g. one-hot encoded matrix of entity extraction).
3. Train a ML model on the data.

My intuition says you must split the data inbetween step 1 and 2, for example, only running TF-IDF or NMF on your training set.

But, I have seen a lot in papers and production, that non-deep learning NLP techniques are often used before a data split.

Answer 1

It is best practice to split the data into train and test datasets. Make modeling choices only on the train data set. Evaluate the usefulness of those choices on the test dataset.

Traditional NLP extraction techniques follow the same logic because they often have modeling choices. One example is the number of topics in non-negative Matrix Factorization (NMF). It is best practice to choose the number of topics on the training dataset, and then evaluate the quality of those topics on the test dataset.

The same logic holds true when estimating a statistic and then making modeling choices on that statistic. Tf–idf (term frequency-inverse document frequency) is a common example. It is best practice to estimate tf-idf on the training set only because later modeling choices are made (or not made) based on tf-idf statistics.