My goal is to train a CNN via transfer learning on a given dataset and to analyze and document the training process. I selected a few CNN architectures and hyperparameters to perform a random search.
For example, some of the hyperparameters are:
- depth of the fully connected network (depth)
- width of the fully connected network (width)
- regularizer function used in the fully connected network (regularizer)
- percentage of trainable layers (train_pcnt)
So my results look similar to this:
| val_acc | architecture | depth | width | regularizer | train_pcnt | ... | | 0.863 | inception_v3 | 2 | 256 | l1 | 0.4 | | | 0.634 | densenet | 1 | 32 | none | 0.8 | | | 0.535 | resnet | 3 | 128 | l1_l2 | 0.5 | |
As a result of the random search, I found a model with which I'm very satisfied, but now I would like to analyze the search space. Ideally, I would be able to make conclusions like
- "The lower the depth of the fully connected network, the better the accuracy.", or
- "L1 regularization performs worse than L2 regularization.",
and so on. However, one can expect that most of these hyperparameters strongly interact with each other. Also, it can be assumed that neither the interactions between the hyperparameters nor the interactions between the hyperparameters and the accuracy are solely linear.
That makes it somewhat difficult to simply stuff all the data into a linear regression and to interpret the coefficients.
Is there still a way to analyze such a high dimensional and complex data with a linear regression? Or do there exist better approaches to perform a basic analysis of the search space?