I know that it is not OK to have too similar data in the train and test set (for example two pictures that differ by only one pixel). I'm trying to find a scientifically valid explanation why it is bad, I mean a paper in a peer-reviewed journal explaining (or even mentioning) this.
Couldn't find anything appropriate for several hours. Do you know any reliable source?
There are several reasons why having too similar data in the train and test set is not recommended. One reason is that it can lead to overfitting, where the model performs well on the training data but poorly on the test data, because it has essentially memorized the training data instead of learning generalizable patterns. This is especially true if the training and test data are very similar, as the model may not be able to generalize to new, unseen data.
Another reason is that it can give an overly optimistic estimate of model performance, because the model is being tested on data that is very similar to the training data. This is not representative of the true performance of the model on new, unseen data.
While there may not be a specific paper addressing the exact scenario you described (two pictures that differ by only one pixel), there are several reasons why having too similar data in the train and test set is generally not recommended in machine learning.
A paper that discusses the importance of representative sampling in machine learning is The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization by Adam Coates, Andrew Y. Ng, and Honglak Lee, published in the Journal of Machine Learning Research (JMLR) in 2011. While this paper does not specifically address the scenario you described, it does emphasize the importance of representative training and test data for accurate machine learning performance evaluation.
A decent reference for this would be Chapter 7 in Elements of Statistical Learning.
For what it's worth, similarity between train and test data isn't a problem in of itself. What you want to ensure is that the test data is representative of a sample of the population you want the model to perform well on. Usually you'll also want it to be "independent" of the training set, so that you are not enforcing similarity between train and test data, thereby overestimating your generalisation performance.
