Hopefully a simple question, but it's a little unclear to me on how best to separate train/validate/test sets.
I have say 100 examples of class A. I'm classifying text into either class A, which I care about, or class B, which could be any text in the world (negative class). I have, obviously, far more examples of class B.
When I split the data into train/validate/test sets, is it imperative that the test set, which is not at all used in training/tuning, NOT have any examples of class A that were used in training? In the real world (and given my limited samples), the text it will classify against will have some exact examples of class A, but not always (there could be variations - of which I do not have all of them).
I can ensure that the test set have unique class B text, but it is unclear if I have to also maintain completely unique class A examples in the test set, since the real world won't necessarily be like this. Would it make sense to also have x% of class A examples from training in the test set, or should it always be 0% in the test set?