Say I have the most basic neural network that performs some task (eg Keras sequential model with one hidden layer, used to binary classification) and a list of ideas how one could improve it (like: add more layers / change size of layers / add regularization / change learning rate / use different batch size and so on).
Because it is for learning purposes, I would like to try different improvements in very structured way, treating each change as an experiment and thoroughly examining results (specified metrics). I'm looking for advice on how to do it in the most robust and at the same time not overly complicated way. Here are my current thoughts:
I should somehow make sure that changes are not due to chance (different random generator state). My idea is to rerun experiment a few times with different seeds and compare statistisc of all runs, not just single set of metrics.
After each change in setup (adding a layer or so) I want to save training logs (I'm using Tensorboard for visualization during training), fitted model and some nicely formatted summary containing metrics table.
After running a few experiments I'll need to find a way to compare results in clear, preferably visual way, so that I can say things like "adding one small layer improved AUC metric on average by 0.05 with std 0.01 compared to baseline model, while increasing the size of existing layer in baseline model didn't have significant effect." Also, it would be nice to have some kind of statistical test to tell if the difference was statistically significant.
I'm waiting for comments about my thoughts and more ideas, even small technical tips like "save models with time index in filename" will be useful :)
