0
$\begingroup$

I have a dataset and would like to train CNNs on subsets of different size of the dataset. I already have a CNN, which classifies very well if I use the entire dataset. Now the question arises if I should really try to additionally optimize the parameters of the CNN for the subsets, regardless of whether I do Data Augmentation or not? Does it really make sense if I try to change the CNN model for the subsets by using RandomizedSearchCV or GridSearchCV to optimize the number of convolutional layers, different learning rates, etc....?

In other words, suppose I found the perfect CNN model for a dataset. Is this model also the perfect model for subsets of this dataset?

I hope someone can give me a hint. For any help, I thank you in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It depends on how you choose the subset of the dataset. One required assumption about data for any Machine Learning model to generalize well on unseen data is that the data must come from the same statistical distribution.

So, if you select the subset at (uniformly) random, it's quite sure that your model will continue to perform well. Otherwise, I think it's better to fine-tune the big model on the subset for better performance.

Improve this answer
$\endgroup$
3
  • $\begingroup$ The subsets are determined using train_test_split with stratify=y and random_state=0. Then the CNN should also work well for the subsets? $\endgroup$
    – Code Now
    Sep 2, 2019 at 12:53
  • $\begingroup$ As far as I understand, stratified sampling is not random sampling, so there should be some difference in performance here. But as a best practice, you should test all your assumption to see if it's true or not. If there's no big jump in accuracy of 2 models, I would say that it's ok to reuse the big model. $\endgroup$
    – TQA
    Sep 2, 2019 at 14:47
  • $\begingroup$ I chose stratify=y to preserve the proportions of the classes in the subsets. According to scikit-learn.org/stable/modules/generated/… train_test_splitsplits arrays or matrices into random train and test subsets. The param shuffle =True by default, so the data should be shuffle before splitting. If this still not random sampling, which method would be better? $\endgroup$
    – Code Now
    Sep 2, 2019 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.