At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, using the same vocabulary created by the training set. They don't contribute any vocabulary, or affect the document frequency (for "term-frequency inverse document frequency" calculation).
However, this is restrictive in a few ways. Firstly, calculating the bag of words model is expensive, and so this prohibits me carrying out k-folds cross validation (since it would require constant re-calculation of the bag of words). My dataset is around 10 million words, and I'm calculating bag of words and bag of bi-grams, which takes around 5 minutes each time.
This also means I currently have holdout data for both my cross validation and test sets, which is data I can't use for training.
Would I be biasing my results significantly if I fit the bag of words on both the training set, and the cross validation set? In other words, if I use the vocabulary in the validation set to calculate the vocabulary for the bag of words? The way I figure, even though they might contribute to the vocabulary, there's no risk of overfitting since the frequency for those specific samples won't be seen at training. This allows me to slice the validation set later however I like, and I still have a "test" set for an accurate predictor of generalisation error (the test set won't be seen at all until test time).
I wonder if there's any precedent for something like this, and what your experiences are doing anything similar.