validation/test set uniqueness question

Question

Hopefully a simple question, but it's a little unclear to me on how best to separate train/validate/test sets.

I have say 100 examples of class A. I'm classifying text into either class A, which I care about, or class B, which could be any text in the world (negative class). I have, obviously, far more examples of class B.

When I split the data into train/validate/test sets, is it imperative that the test set, which is not at all used in training/tuning, NOT have any examples of class A that were used in training? In the real world (and given my limited samples), the text it will classify against will have some exact examples of class A, but not always (there could be variations - of which I do not have all of them).

I can ensure that the test set have unique class B text, but it is unclear if I have to also maintain completely unique class A examples in the test set, since the real world won't necessarily be like this. Would it make sense to also have x% of class A examples from training in the test set, or should it always be 0% in the test set?

Answer 1

Let's take your example. If you have 2 classes A and B. The percentage of class A present in the data is less than B. So basically you have an imbalanced dataset. You have to ensure (and this goes for both balanced and imbalanced datasets) that both the train and test sets contain both A and B classes.

You ask weather is it ok to not have any data of class B in the test set and I would say it is wrong. You have to have both the classes in the test and in the train/validation set. To achieve this, you can use stratify = target in the train_test_split function you use when splitting the data. Also you should use nested cross validation to ensure all your data is used.

Keep in mind that stratify = target won't solve your class imbalance problem and you have to deal with it separately. It only ensures that the both the classes are distributed both in the train/valid and test sets and hence you get a fair distribution. Now how you deal with it depends on you.

EDIT 1: Based on the comment the SO is asking weather he can use the samples that were in the train set in the test set as well. Then the answer is pretty simple. A big NO! You cannot use the data that was in your train set into your test set as it will lead to data leakage!! This is a big mistake and would result in overly positive results. Google data leakage and you'll see how big of a problem it is. Basically you cannot use data that is present in the train set into your test set.

Keep you test data completely and absolutely separate from your train/valid set. Only then you can get a generalizable result that would benefit you in the real world application.

As I mentioned before, if you have a small dataset, use nested cross validation. It will select the best model and tune your Hyperparameters at the same time and ensure that all you data gets utilized.

Answer 2

Of your 100 positive text examples, you want to split that into NON OVERLAPPING train, validate & test (say 75%, 5% & 20% respectively) & likewise randomly sample same quantity of negative examples for each split to keep each balanced.

Now, if your data naturally lends itself to many dupes you can either keep the dupes across & within your splits as this proxies the distributions the model will see in production. However, since you have very limited data, as a technique to increase variance, you can repeatedly swap dupes within each split with dupes from other splits until you minimize the dupe rates within each split.

Another trick for binary classification on tiny data is to sample a little bit more of your plentiful class to provide more examples. The idea is if the model can reliably detect one class, you can infer the other. Note that this will unbalance the data but it might be tolerable if you keep it say at a 48%/52% ratio.

At these small dataset sizes, also look into labeling more or somehow synthesizing new examples (ie: synonyms, negations, etc) or even rely on leave-one out-cross-validation to evaluate an estimate while still training the real model with 100% of your tiny dataset.