My goal is to train a CNN via transfer learning on a given dataset and to analyze and document the training process. I selected a few CNN architectures and hyperparameters to perform a random search.

For example, some of the hyperparameters are:

  • depth of the fully connected network (depth)
  • width of the fully connected network (width)
  • regularizer function used in the fully connected network (regularizer)
  • percentage of trainable layers (train_pcnt)
  • ...

So my results look similar to this:

| val_acc | architecture | depth | width | regularizer | train_pcnt | ... |
| 0.863   | inception_v3 | 2     | 256   | l1          | 0.4        |     |
| 0.634   | densenet     | 1     | 32    | none        | 0.8        |     |
| 0.535   | resnet       | 3     | 128   | l1_l2       | 0.5        |     |

As a result of the random search, I found a model with which I'm very satisfied, but now I would like to analyze the search space. Ideally, I would be able to make conclusions like

  • "The lower the depth of the fully connected network, the better the accuracy.", or
  • "L1 regularization performs worse than L2 regularization.",

and so on. However, one can expect that most of these hyperparameters strongly interact with each other. Also, it can be assumed that neither the interactions between the hyperparameters nor the interactions between the hyperparameters and the accuracy are solely linear.

That makes it somewhat difficult to simply stuff all the data into a linear regression and to interpret the coefficients.

Is there still a way to analyze such a high dimensional and complex data with a linear regression? Or do there exist better approaches to perform a basic analysis of the search space?

  • 1
    $\begingroup$ One simple thing to consider is just to plot the univariate results (model score vs. one hyperparameter). You'll see lots of vertical noise due to the unplotted hyperparameters, but if you have enough random points you should still be able to see trend. And you can extend this to bivariate 3D plots to see some interaction effects. But I'll look forward to a more statistical / rigorous / automatic Answer. $\endgroup$ – Ben Reiniger Aug 26 '19 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.