Is it valid to include your validation data in your vocabulary for NLP?

Question

At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, using the same vocabulary created by the training set. They don't contribute any vocabulary, or affect the document frequency (for "term-frequency inverse document frequency" calculation).

However, this is restrictive in a few ways. Firstly, calculating the bag of words model is expensive, and so this prohibits me carrying out k-folds cross validation (since it would require constant re-calculation of the bag of words). My dataset is around 10 million words, and I'm calculating bag of words and bag of bi-grams, which takes around 5 minutes each time.

This also means I currently have holdout data for both my cross validation and test sets, which is data I can't use for training.

Would I be biasing my results significantly if I fit the bag of words on both the training set, and the cross validation set? In other words, if I use the vocabulary in the validation set to calculate the vocabulary for the bag of words? The way I figure, even though they might contribute to the vocabulary, there's no risk of overfitting since the frequency for those specific samples won't be seen at training. This allows me to slice the validation set later however I like, and I still have a "test" set for an accurate predictor of generalisation error (the test set won't be seen at all until test time).

I wonder if there's any precedent for something like this, and what your experiences are doing anything similar.

Answer 1

Yes. That should be fine. What you suggest makes sense and should not bias the results. The reasons you give are good ones. For any reasonable classifier, if the value of an attribute is always zero in the training set, that should cause the attribute to be essentially ignored.

There is a simple test to let you confirm this. You can try, for each document in the validation set, zeroing out the entries in the feature vector that correspond to words that were not present in any document in the training set, and see if that changes the classification. If it doesn't, then you know that your method has had no effect and hasn't introduced any bias.

---

Here is another alternative, if you prefer:

Conceptually, in an ideal world the vocabulary for each fold would only include words in the training set.

As a matter of implementation, it's certainly possible to implement that in a more efficient way than re-generating the vocabulary and re-generating the feature vectors for each fold. As long as your implementation generates the same vectors, it doesn't matter how it obtains them.

Thus, you could have an implementation that works like the following:

• Generate a "superset vocabulary" that includes all of the words found anywhere in the dataset (not just the training set).

• Generate a sparse feature vector for each document, based on the "superset vocabulary". I suggest that you represent these in an efficient way, e.g., as a Python dictionary that maps from word to count.

• For each fold:

  • Split into training and validation sets.

  • Generate a subset vocabulary, containing only the words in the training set. Probably there is a small set $S$ of words that are in the superset vocabulary but not the subset vocabulary, so I suggest storing that set $S$.

  • For each document in the training set and validation set, generate a derived feature vector for the document, using only the subset vocabulary. I suggest generating this by starting from the feature vector for the superset vocabulary, then removing the words in $S$. This should be more efficient than regenerating the feature vector from scratch.

  • Now train and validate these derived feature vectors.

This is equivalent to the "conceptually ideal" approach, but will run faster.

That said, I think your proposal is the pragmatic one, and in practice should yield equivalent results, and be even faster still.