I want to know if we can change the structure of the feature-extraction layers in the deep networks architectures, for example can we add more Rectified Linear Unit (ReLU) activation function or is it the same sequence that we should respect ?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You can change whatever you like!
The benefits will depend on your data and what exactly you are comparing to. As you didn't say exactly what sequence you have as your starting point, I can't compare anything exactly, but provide a simple outline.
Generally, you follow the following recipe:
Input layer: perform any reshaping or normalisation of your data
Multiplication layer: either fully-connected layer, a convolutional layer or something else
- Non-linearity:
relu,tanh- gives the network its power to fit non-linear functions - regularisation layer:
batchnorm,dropout- help models converge and prevent overfitting
This scheme is then repeated as many times as you feel is necessary.
Doing something like applying several ReLU layers, one after another, definitely doesn't make sense, as it would actually change anything! A ReLU simply filters out negative values, setting them to zero. Applying it a second time would therefore have no effect.
