I'm trying to implement a simple text classifier wherein the data is split into training (70%) and testing (30%) sets, but cross validation (k=10) to be performed on the training set.
My main concern here is the basis used for transforming a given set. I've seen tutorials where the whole data set was used to fit a Count/TFIDF vectorizer, but wouldn't this introduce bias when transforming the validation and testing sets since the previously mentioned sets were included in the whole data set? Or is the bias so small that it is acceptable to do so?
Within a fold, would it be better if the training set was used to fit a vectorizer and transform the validation set? And for testing, should the training + validation sets be used to fit a vectorizer to transform the testing set?
And on this note, should the validation set also be treated as "unseen" data similar to the test set?
A lot of the tutorials and notes online show ready made datasets, but there are certain cases with text data where values are dependent on a given set of documents (e.g. like how IDF is computed based on some group of documents) and that there are a lot more nitty gritty details when extracting features.
I think I'm just confused and am seeking a little clarification regarding this manner. Thanks!
