I have a dataset containing images taken under 4 different conditions. When training the model, I use the same proportion of images (25%) from each condition. Then, I'm testing on 4 different test sets where each set corresponds to a particular condition.
I noticed that the model is doing well on two of the test sets (above 95%) and bad on the other two (below 50%). Now, I'm thinking maybe this has to do with the fact that I'm splitting the training and validation randomly (70% training vs. 30% validation), and I should probably devise a better CV scheme, but not sure if the problem comes form there.
There is another detail which may be relevant (although I don't think so because I'm using ResNet18 and pooling layers should take care of this issue) is that the shapes I'm trying to detect exit in the same location (say upper right) for the two first test datasets (similar to the training) and on the bad test datasets, the shapes exist in a different location (say lower left).