This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction that is "promising". In that regard, I hope that someone can give general hints and recommendations, even though there certainly will not be a single, perfect answer.
I'm currently assigned a rather vague research task, along the lines of "Here's some data, throw it on some neural network, and see what you can figure out". And the main problem that I'm currently facing is the parameter space.
As a specific (but somewhat arbitrary) example, consider a Python/Keras environment, particularly the stacked LSTM example from the keras documentation.
The degrees of freedom here are, broadly speaking,
- The number of layers
- The type of layers, which may be
- Core layers, ....
- Convolutional Layers,...
- Pooling layers, ...
- Locally connected layers, ...
- Embedding layers, ...
- Merge layers, ...
- Advanced activation layers, ...
- Normalization layers, ...
- Noise layers, ...
- Layer wrappers, ...
- Custom layers,...
- Recurrent layers, which may be
- RNN, ...
- SimpleRNN, ...
- GRU, ...
- ConvLSTM2D, ...
- ConvLSTM2DCell, ...
- SimpleRNNCell, ...
- ConvLSTM2DCell, ...
- GRUCell, ...
- LSTMCell, ...
- CuDNNGRU, ...
- CuDNNLSTM, ...
- LSTM, which has the parameters
- units, ...
- recurrent_activation, ...
- use_bias, ...
- kernel_initializer, ...
- recurrent_initializer, ...
- bias_initializer, ...
- unit_forget_bias, ...
- kernel_regularizer, ...
- recurrent_regularizer, ...
- bias_regularizer, ...
- activity_regularizer, ...
- kernel_constraint, ...
- recurrent_constraint, ...
- bias_constraint, ...
- dropout, ...
- recurrent_dropout, ...
- implementation, ...
- return_sequences, ...
- return_state, ...
- go_backwards, ...
- stateful, ...
- unroll, ...
- activation, which can be one of
- softmax, ...
- elu, ...
- selu, ...
- softplus, ...
- softsign, ...
- tanh, ...
- sigmoid, ...
- hard_sigmoid, ...
- exponential, ...
- linear, ...
- sigmoid, ...
- relu, which has the parameters
- alpha
- max_value
- threshold
- The compilation step, which has parameters
- metrics: ...
- loss_weights: ...
- sample_weight_mode: ...
- weighted_metrics: ...
- target_tensors: ...
- loss, which may be one of 14 functions, each with several parameters
- optimizer, which may be one of
- SGD, ...
- RMSprop, ...
- Adagrad, ...
- Adadelta, ...
- Adamax, ...
- Nadam, ...
- Adam, which has parameters with
- lr, with a default value of 0.002
- beta_1, with a default value of 0.9
- beta_2, with a default value of 0.999
- epsilon, with a default value of None
- schedule_decay, with a default value of 0.004
- The fitting/training step, which has parameters
- training/target data: ...
- validation_split/validation_data: ...
- shuffle: ...
- class_weight: ...
- sample_weight: ...
- initial_epoch: ...
- steps_per_epoch: ...
- validation_steps: ...
- validation_freq: ...
- epochs:...
- batch_size,...
Or to put it that way: The example from the documentation is based on one point inside an infinite parameter space, and the choice of this point seeems to be disturbingly arbitrary.
I know that it is "simple" to get started in some way, because one can just copy+paste the example, change the input dat and google error messages until it "runs", and then hope that the default parameter values are sensible and start to tweak the remaining parameters to see how the results change. But this is not research, and not science, and in no way productive.
And of course, one can argue about each and every point. One can look up the meaning of the beta_2 parameter of the adam optimizer of the training process in a 20-page, math-heavy research paper. More generally, there are some resources about hyperparameter tuning and of course some stackexchange network Q/As that either try to explain a parameter, or link to the papers that explain them.
But there is no way of really understanding the effects of these parameters. Even the effects of single, seemingly simple parameters (like the batch size in the link above) involve mathematical models that cannot be understood, but only "examined", and even then, conclusions can only be made for varying one (or few) parameters and leaving hundreds of other parameters fixed.
Recently, the idea of explainable artificial intelligence has gained some attention - probably also as a result of identifying the complexity of such systems as a severe problem. But until now, we have Keras and other high-level APIs that raise some of these core questions:
How does one decide which which of the parameters in the (virtually infinite) parameter space are worth tweaking?
On the highest level, there may be rules of thumb. On the one hand, there are characteristics of the input data. For example, the dimensionality or structure of the data (e.g. whether it's point data or time series). On the other hand, there are certain analysis goals (e.g. prediction or anomaly detection). These will narrow down the parameter space in terms of numbers of layers or types of layers that "could make sense" or are "usually applied to this kind of problem". But it still feels like guesswork for me.
How can one profoundly argue that a certain parameter does not have to be tweaked?
If someone finds a network layout or a combination of parameters that "works well" for a certain data set and mining task, then everything is fine. But assuming that the results are not as desired, how can one counter the argument that it "might work if you tweaked this-and-that parameter"? Again, the only way to counter such an argument could be to empirically examine this single parameter (while leaving all others fixed) and then draw some handwaving conclusions from the observed effects (which almost have to be unjustified in view of the complexity of the underlying system).
(A shorter and more provocative way of phrasing the question could be: Is it right that data science with neural networks is just trial-and-error?, but I'm afraid that I already know the answer to that one...)
