CNNs - Hyperparameter tuning with different training sizes of the same data set

Question

I would like to compare how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set. For this I would like to use a data set like MNIST or Fashion MNIST.

I would like to start by first training a CNN with only a subset ob about 1000 images, then a CNN with a subset ob about 5000 images, then with a subset of about 10000 images etc.

For each ot these data sets an optimized CNN should be generated.

My current plan would be the following:

1. Create a simple base model with only 1 Conv Layer (with 32 filter), MaxPooling2D, Dropout (0.5) and a Dense Layer with about 100 Neurons.
2. Using this base model, I would search for the smallest dataset (with 1000 images) using GridSearchCV for batch size and optimizer und reuse both for all other datasets.

3. Then I would do the following for each data set:

   • Search for the optimal architecture of the network (Number of Conv layer, Number of filter, Number of dense layer, Number of Neurons in Dense layer) using GridSearchCV (RandomizedSearchCV will propably be less suitable at this point)

   • Search for learning rate, weight decay, dropout rate etc. (using GridSearch or RandomizedSearch).

Maybe I could just use the default setting for one or the other not so important parameter and exclude them from the search.

I am not sure if my approach is really recommendable.

I also tend to use RandomizedSearchCV to determine all hyperparameters simultaneously for each data set.

Many thanks in advance for your comments and suggestions.

Answer 1

First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it.

Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can easily take hundreds of thousands or millions of trainable parameters. Running cross validation on those could be computationally impossible.

In other words, keep the two problems separated.

I suggest you to take a look at common CNN architectures. There's plenty of them on Kaggle competitions (for MNIST, Fashion MNIST or any other dataset). You can choose one that works fine, and then use it to show how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set.