I would like to compare how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set. For this I would like to use a data set like MNIST or Fashion MNIST.
I would like to start by first training a CNN with only a subset ob about 1000 images, then a CNN with a subset ob about 5000 images, then with a subset of about 10000 images etc.
For each ot these data sets an optimized CNN should be generated.
My current plan would be the following:
- Create a simple base model with only 1 Conv Layer (with 32 filter), MaxPooling2D, Dropout (0.5) and a Dense Layer with about 100 Neurons.
- Using this base model, I would search for the smallest dataset (with 1000 images) using GridSearchCV for batch size and optimizer und reuse both for all other datasets.
Then I would do the following for each data set:
Search for the optimal architecture of the network (Number of Conv layer, Number of filter, Number of dense layer, Number of Neurons in Dense layer) using GridSearchCV (RandomizedSearchCV will propably be less suitable at this point)
Search for learning rate, weight decay, dropout rate etc. (using GridSearch or RandomizedSearch).
Maybe I could just use the default setting for one or the other not so important parameter and exclude them from the search.
I am not sure if my approach is really recommendable.
I also tend to use RandomizedSearchCV to determine all hyperparameters simultaneously for each data set.
Many thanks in advance for your comments and suggestions.