1
$\begingroup$

I just finished reading Delving Deep into Rectifiers paper. This paper proposes a new activation function called PReLU. Maybe it is obvious, because the paper did not mention it, but I want to know when is the parameter of a PReLU updated? Is it updated before weight update or after weight update?

$\endgroup$
1
$\begingroup$

Is it updated before weight update or after weight update?

It doesn't usually matter, provided your NN framework has separate gradient calculation stage from parameter update stage.

The gradients should be collected with all weights and parameters at their current values (during back-propagation). Then the parameters are updated, using those gradients. There should be no interaction between the parameters during the update step.

However, if you have implemented the training code yourself from scratch, and have interleaved the gradient calculations and weight updates, then this only works in a pure online situation with weights updated as the gradients are calculated. In that scenario, you should probably update the weights for a layer before updating the PReLU parameter for the same layer, to prevent changes in the activation function altering the gradients during your calculations (this is not a concern with usual weight updates).

Alternatively, separate your gradient calculations from the weight update steps. This is more flexible and allows you to use other more advanced optimisations and layer designs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.