Now the challenge becomes creating a training dataset that is varied enough to ensure good generalization. To go back to the dog example, I would not want to train my model using only pictures where the dog appears in the lower left corner!

This is a great example!

Maybe I did not phrase my question properly. Imagine an algorithm aiming to identify a given target (e.g. a dog) from a series of photos. Would it make sense to use the position of the dog in each picture as a feature?

Thanks for your answer, @Ipounng! You make a valid point: I know regularization from geophysical data inversion and it does indeed prevent overfitting by promoting a certain behavior (e.g. smoothness or "blockyness"). In that regards, you are right that constraining the model prevents it from overfitting.

Thanks for your feedback! Were you able to implement this strategy? How do the results compare to those obtained using Shao's method?

Thanks for your reply, Pedro. I confirm that there is no data leakage from the bigger dataset to the training dataset. I will give k-fold cross-validation a shot!