1
$\begingroup$

i.e. if a model with 10 inputs, say,

model = Sequential()
model.add(Dense(units=10, activation='relu'))
model.add(Dense(units=10, activation='relu'))
model.add(Dense(2, activation='relu'))
optimizer=tf.keras.optimizers.SGD(learning_rate=0.01)
model.compile(loss='mse', optimizer=optimizer, metrics=[tf.keras.metrics.MeanSquaredError()])

If it can't capture the desired relationship, we may perturb many things, e.g. a random search CV (such as learning rate and activation functions).

Is it wise to have more units than first-layer inputs, or should we seek to add more layers if we require more 'complexity'?

$\endgroup$

1 Answer 1

2
$\begingroup$

This is essentially a difficult, unsolved question in deep learning for which there is no unified answer, yet. This is because firstly the architecture of a network has "uncountably" many degrees of freedom and secondly because the mathematical foundations are not developed enough in some domains.

Now a little more concrete:

  1. If you have time to experiment, you could for instance do a small study:
  • one model with few large layers
  • one model with many narrow layers

The important point is that both models should have approximately the same number of parameters. Test the performance and compare.

  1. In domains like image recognition where CNNs are used, there seems to be a tendency to use more layers with smaller kernels, but many channels. Smaller kernels decrease the number of parameters and larger channels increase the number of parameters. Smaller kernels essentially mean less multiply-adds. Equivalently for linear fully-connected layers, this would mean less neurons per layer, because this means less multiply-adds. Also keep in mind; with increasing number of neurons per layer, the matrices that need to be multiplied grow exponentially. So, it may be computation-wise smarter to start out with more linear layers with fewer neurons than vice versa. In all of the experiments, start with a very tiny (like, really tiny) model that underfits and then gradually increase the complexity of the model. This takes some experiemental effort. Do not start with an overly large model that takes hours to compute!
$\endgroup$
2
  • $\begingroup$ V informative. Thanks! $\endgroup$
    – Socorro
    Commented Apr 7, 2022 at 12:04
  • $\begingroup$ It would be very nice if you could link your sources. $\endgroup$
    – MaxPowers
    Commented Jun 19 at 5:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.