2
$\begingroup$

This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction that is "promising". In that regard, I hope that someone can give general hints and recommendations, even though there certainly will not be a single, perfect answer.

I'm currently assigned a rather vague research task, along the lines of "Here's some data, throw it on some neural network, and see what you can figure out". And the main problem that I'm currently facing is the parameter space.

As a specific (but somewhat arbitrary) example, consider a Python/Keras environment, particularly the stacked LSTM example from the keras documentation.

The degrees of freedom here are, broadly speaking,

  • The number of layers
  • The type of layers, which may be
    • Core layers, ....
    • Convolutional Layers,...
    • Pooling layers, ...
    • Locally connected layers, ...
    • Embedding layers, ...
    • Merge layers, ...
    • Advanced activation layers, ...
    • Normalization layers, ...
    • Noise layers, ...
    • Layer wrappers, ...
    • Custom layers,...
    • Recurrent layers, which may be
      • RNN, ...
      • SimpleRNN, ...
      • GRU, ...
      • ConvLSTM2D, ...
      • ConvLSTM2DCell, ...
      • SimpleRNNCell, ...
      • ConvLSTM2DCell, ...
      • GRUCell, ...
      • LSTMCell, ...
      • CuDNNGRU, ...
      • CuDNNLSTM, ...
      • LSTM, which has the parameters
        • units, ...
        • recurrent_activation, ...
        • use_bias, ...
        • kernel_initializer, ...
        • recurrent_initializer, ...
        • bias_initializer, ...
        • unit_forget_bias, ...
        • kernel_regularizer, ...
        • recurrent_regularizer, ...
        • bias_regularizer, ...
        • activity_regularizer, ...
        • kernel_constraint, ...
        • recurrent_constraint, ...
        • bias_constraint, ...
        • dropout, ...
        • recurrent_dropout, ...
        • implementation, ...
        • return_sequences, ...
        • return_state, ...
        • go_backwards, ...
        • stateful, ...
        • unroll, ...
        • activation, which can be one of
          • softmax, ...
          • elu, ...
          • selu, ...
          • softplus, ...
          • softsign, ...
          • tanh, ...
          • sigmoid, ...
          • hard_sigmoid, ...
          • exponential, ...
          • linear, ...
          • sigmoid, ...
          • relu, which has the parameters
            • alpha
            • max_value
            • threshold
  • The compilation step, which has parameters
    • metrics: ...
    • loss_weights: ...
    • sample_weight_mode: ...
    • weighted_metrics: ...
    • target_tensors: ...
    • loss, which may be one of 14 functions, each with several parameters
    • optimizer, which may be one of
      • SGD, ...
      • RMSprop, ...
      • Adagrad, ...
      • Adadelta, ...
      • Adamax, ...
      • Nadam, ...
      • Adam, which has parameters with
        • lr, with a default value of 0.002
        • beta_1, with a default value of 0.9
        • beta_2, with a default value of 0.999
        • epsilon, with a default value of None
        • schedule_decay, with a default value of 0.004
  • The fitting/training step, which has parameters
    • training/target data: ...
    • validation_split/validation_data: ...
    • shuffle: ...
    • class_weight: ...
    • sample_weight: ...
    • initial_epoch: ...
    • steps_per_epoch: ...
    • validation_steps: ...
    • validation_freq: ...
    • epochs:...
    • batch_size,...

Or to put it that way: The example from the documentation is based on one point inside an infinite parameter space, and the choice of this point seeems to be disturbingly arbitrary.


I know that it is "simple" to get started in some way, because one can just copy+paste the example, change the input dat and google error messages until it "runs", and then hope that the default parameter values are sensible and start to tweak the remaining parameters to see how the results change. But this is not research, and not science, and in no way productive.

And of course, one can argue about each and every point. One can look up the meaning of the beta_2 parameter of the adam optimizer of the training process in a 20-page, math-heavy research paper. More generally, there are some resources about hyperparameter tuning and of course some stackexchange network Q/As that either try to explain a parameter, or link to the papers that explain them.

But there is no way of really understanding the effects of these parameters. Even the effects of single, seemingly simple parameters (like the batch size in the link above) involve mathematical models that cannot be understood, but only "examined", and even then, conclusions can only be made for varying one (or few) parameters and leaving hundreds of other parameters fixed.


Recently, the idea of explainable artificial intelligence has gained some attention - probably also as a result of identifying the complexity of such systems as a severe problem. But until now, we have Keras and other high-level APIs that raise some of these core questions:

How does one decide which which of the parameters in the (virtually infinite) parameter space are worth tweaking?

On the highest level, there may be rules of thumb. On the one hand, there are characteristics of the input data. For example, the dimensionality or structure of the data (e.g. whether it's point data or time series). On the other hand, there are certain analysis goals (e.g. prediction or anomaly detection). These will narrow down the parameter space in terms of numbers of layers or types of layers that "could make sense" or are "usually applied to this kind of problem". But it still feels like guesswork for me.

How can one profoundly argue that a certain parameter does not have to be tweaked?

If someone finds a network layout or a combination of parameters that "works well" for a certain data set and mining task, then everything is fine. But assuming that the results are not as desired, how can one counter the argument that it "might work if you tweaked this-and-that parameter"? Again, the only way to counter such an argument could be to empirically examine this single parameter (while leaving all others fixed) and then draw some handwaving conclusions from the observed effects (which almost have to be unjustified in view of the complexity of the underlying system).


(A shorter and more provocative way of phrasing the question could be: Is it right that data science with neural networks is just trial-and-error?, but I'm afraid that I already know the answer to that one...)

$\endgroup$
1
$\begingroup$

I am afraid there is no real answer to your question. But to still answer it I can only advise you to look for practical articles relating to your specific problem. May I suggest you to start with this very general article by LeCun : http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf ? Citing the introduction directly :

"Designing and training a network using backprop requires making many seemingly arbitrary choices [...] These choices can be critical, yet there is no foolproof recipe for deciding them because they are largely problem and data dependent. However, there are heuristics and some underlying theory that can help guide a practitioner to make better choices."

| improve this answer | |
$\endgroup$
  • $\begingroup$ From a first, quick (!) glance (it has 44 pages...), it seems to cover the "high level" parameters that I mentioned under the first headline, but also only on a methodological level (e.g. describing the advantage of batches, but no guideline of whether a batch size of 4,8,16 or 1024 should be chosen - similarly, for all the other (numerical) parameters that are hidden deeply within 5 lines of keras code). Also, it's from '98, and I think the problem amplified with with the advent of "deep learning". But I'll have to take a closer look at that - thanks for the pointer! $\endgroup$ – Marco13 Jan 24 at 15:27
  • $\begingroup$ Most of the parameters of the NN you mention seems to fall under the following remarks of the paper : "there is no foolproof recipe for deciding them because they are largely problem and data dependent". And there is, in my opinion two categories of them : those you can avoid by intuition (this was designed to deal with X problem, which I don't have so I shouldn't use it), those you can't avoid but need to tune manually (with some intuition...). So you are mostly left with tuning the calibration process. $\endgroup$ – lcrmorin Jan 24 at 15:40
  • $\begingroup$ Btw, I found this paper quite up to date, as backpropagation hasn't changed much since 1998 and really usefull (I have to concede that I found some mistakes I made after reading this). More generally, it helps understand what type of problems you can encounter during calibration and deal with them. $\endgroup$ – lcrmorin Jan 24 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.