2
$\begingroup$

Suppose a neural network is built for a binary classification problem such as recognize the face as a smiley face or not, by using a dataset of 1000 persons and each person has ten images of his face. If the dataset randomly spilt into trainset and testset by a ratio of 70:30, in this case, there is a big chance face image of same persons will be used in both the trainset and testset, so is this considered to be data leakage (train-test contamination)?

$\endgroup$
0
$\begingroup$

Yes, this is a form of data leakage. The testing data should not be linked to the training data in any way.

Another way to think of it is, if someone were to try replicating your results with their own test set, would your test set have given you an advantage such that your results are generally better than theirs?

$\endgroup$
6
  • $\begingroup$ if my testset has better accuracy than others that means my testset is biased in someway $\endgroup$
    – AI_new2
    Jul 19 '20 at 22:47
  • $\begingroup$ I saw one researcher do that(use the same person in train&test), I told him that wrong, but he insisted that every image independent from each other regardless of belong to the same person, give an example of using same human race in training&testing would be data leakage also! $\endgroup$
    – AI_new2
    Jul 19 '20 at 22:50
  • $\begingroup$ You are right! (and he is wrong). Yet another way to picture this problem is with synthetic/augmented data. If, for example, you flip images so that you double the number of instances in your dataset, you must keep both flipped instances together when splitting the data. If you didn't do this, then the testing data would have (transformed) training data in it. Of course, this is a very clear transformation (reflection), but taking images from multiple angles or having the same person do multiple expressions is similarly a transformation of the training set. $\endgroup$ Jul 19 '20 at 22:55
  • $\begingroup$ A simpler example would be binary classification of handwritten digits, limited to ones and zeros. If you have the same person write a lot of ones and zeros, you couldn't mix them into training and testing data. Even if every image was technically unique, it is still introducing bias in the testing set. The same principle applies to faces, even though those images are more complicated. $\endgroup$ Jul 19 '20 at 22:59
  • $\begingroup$ Just curious, what would happen if the same face(s) were also in the validation data or later datasets? Would the validation data score tend to be higher in that case? $\endgroup$
    – Donald S
    Jul 20 '20 at 6:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.