I want to know if we can change the structure of the feature-extraction layers in the deep networks architectures, for example can we add more Rectified Linear Unit (ReLU) activation function or is it the same sequence that we should respect ?


1 Answer 1


You can change whatever you like!

The benefits will depend on your data and what exactly you are comparing to. As you didn't say exactly what sequence you have as your starting point, I can't compare anything exactly, but provide a simple outline.

Generally, you follow the following recipe:

  • Input layer: perform any reshaping or normalisation of your data

  • Multiplication layer: either fully-connected layer, a convolutional layer or something else

  • Non-linearity: relu, tanh - gives the network its power to fit non-linear functions
  • regularisation layer: batchnorm, dropout - help models converge and prevent overfitting

This scheme is then repeated as many times as you feel is necessary.

Doing something like applying several ReLU layers, one after another, definitely doesn't make sense, as it would actually change anything! A ReLU simply filters out negative values, setting them to zero. Applying it a second time would therefore have no effect.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.