What is best practice for applying traditional NLP extraction techniques a pre-processing for ML models?

Given a pipeline:

  1. Collect raw data.
  2. Parse full data set with a variety of traditional NLP techniques, to create model-compatible features (e.g. one-hot encoded matrix of entity extraction).
  3. Train a ML model on the data.

My intuition says you must split the data inbetween step 1 and 2, for example, only running TF-IDF or NMF on your training set.

But, I have seen a lot in papers and production, that non-deep learning NLP techniques are often used before a data split.


1 Answer 1


It is best practice to split the data into train and test datasets. Make modeling choices only on the train data set. Evaluate the usefulness of those choices on the test dataset.

Traditional NLP extraction techniques follow the same logic because they often have modeling choices. One example is the number of topics in non-negative Matrix Factorization (NMF). It is best practice to choose the number of topics on the training dataset, and then evaluate the quality of those topics on the test dataset.

The same logic holds true when estimating a statistic and then making modeling choices on that statistic. Tf–idf (term frequency-inverse document frequency) is a common example. It is best practice to estimate tf-idf on the training set only because later modeling choices are made (or not made) based on tf-idf statistics.

  • $\begingroup$ Thanks Brian, this confirms my suspicions and will accept the answer. In your experience, have you come across any NLP extraction methods that wouldn't need to be applied on a dataset before it's split? $\endgroup$
    – GooJ
    Commented Sep 8, 2022 at 15:09
  • 1
    $\begingroup$ I treat every option at every step (e.g., sentence segmentation, tokenization, stemming, ...) as a hyperparameter to be optimized so I apply train-test split to everything. $\endgroup$ Commented Sep 8, 2022 at 15:13
  • $\begingroup$ Perfect, thank you - have you come across this done incorrectly in production, or am I just getting unlucky? $\endgroup$
    – GooJ
    Commented Sep 8, 2022 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.