3
$\begingroup$

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used.

When we talk about solving the vanishing gradient problem in RNN, we use a more complex architecture (e.g. LSTM). In both of these, activation function is tanh. Can't we use ReLU instead of tanh in RNNs for solving vanishing gradients too rather than opting for a more complex architecture?

$\endgroup$
3
$\begingroup$

Can't we use ReLU instead of tanh in RNNs for solving vanishing gradients too rather than opting for a more complex architecture?

Yes you can. I'd expect it to help a little with vanishing gradients, but not be as effective generally as a LSTM is. Also, without some kind of bounded function in the feedback loop, a basic RNN using ReLU will suffer very easily from exploding values, resulting in very large errors as the feedback is run over multiple time steps - so you may need some adjustment like gradient clipping to account for that.

If you already have a problem to analyse, have prepared your data and decided on a top level architecture (e.g. how many recurrent layers, whether this is a seq2seq type model etc), then it should be relatively easy to compare LSTM with a basic RNN using ReLU. It is likely different hyperparameters would suit each architecture so you may need to do a search, and you could compare best possible results for each model, or look at things such as number of parameters or training time.

I expect many researchers have already tried this comparison, although the examples I found on a quick search appeared to use tanh for the simple RNN.

Even using ReLU in simple RNN architecture, I suspect that by most measures and in most problems, the LSTM or GRU architectures will perform better. There will be some edge cases where RNN performs better, such as if your goal is to get to a certain accuracy quickly during training and not necessarily to get the best accuracy in the end.

$\endgroup$
0
$\begingroup$

RELU can only solve part of the gradient vanishing problem of RNN because the gradient vanishing problem is not only caused by activation function.

enter image description here

enter image description here

equal to

enter image description here

see above function, the hidden state derivative will depend on both activation and Ws, if Ws's max eigen value < 1, the long term dependency's gradient will be vanished.

check the chain multiplication section, the Ws will be multiplied multiple times

$\endgroup$
1
  • 1
    $\begingroup$ To combat this problem, ReLU activated RNN is usually initialized with orthogonal Ws. If all of the final Ws singular values is less than 1, a little bit of doubt should be cast about the necessity of long term dependency upon the task at hand: openreview.net/forum?id=Hygxb2CqKm $\endgroup$
    – Minh Khôi
    Jul 10 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.