I have a dataset and would like to train CNNs on subsets of different size of the dataset. I already have a CNN, which classifies very well if I use the entire dataset. Now the question arises if I should really try to additionally optimize the parameters of the CNN for the subsets, regardless of whether I do Data Augmentation or not? Does it really make sense if I try to change the CNN model for the subsets by using RandomizedSearchCV or GridSearchCV to optimize the number of convolutional layers, different learning rates, etc....?
In other words, suppose I found the perfect CNN model for a dataset. Is this model also the perfect model for subsets of this dataset?
I hope someone can give me a hint. For any help, I thank you in advance.