I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.).
I do not modify the ML model between the different dataset training.
Should I use the same Test dataset to compare them ? This dataset would not be changed through the testing and would contain data that can't be found else where. It would not be more suited for a specific training dataset.
Or should I use a Test dataset that has the same caracteristic as the Training Dataset ? So that I can compare them at their best ?
For example, if I want to compare the dataset A and B, should I use a combinaison of Test dataset A and Test dataset B ? or When testing dataset A, use Test dataset A and when testing dataset B, use Test dataset B.