0
$\begingroup$

Looking at the literature, there are 2 distinct approaches to LSTM


Some people use recurrent weights with Input, Forget, Output - notice, their equations don't even mention dataGate, they start from describing the $f$ or $i$ gate (1), Wikipedia: (2)

Lke this:

enter image description here


Other people use recurrent weights with dataGate (z), Input, Forget, Output (1) (2) (3), (4)

Like this:

enter image description here

Having tried the first approach (I still used standard weights on dataGate but no recurrent weights in that place) seems fine but a little odd - network converges well, but very rarely jumps over local mimima, as if too high learning rate.

Personally I like the second approach (recurrent weights on all 4 entry points), which one do I use?


Edit:

to make things worse, the "LSTM Peephole paper", page 121 describes 3 connections (no peephole for dataGate).

This destroys the equal-sizes of my matrices, because now 3 gates use the Usual, Recurrent and Peephole weights, but the very first gate only uses the Usual weights

$\endgroup$
1
$\begingroup$

There's more than two or three variants with regards to LSTM.

A paper that explores these variants can be read here:

LSTM:A Search Space Odyssey

$\endgroup$
1
$\begingroup$

Actually, both of the examples in the question were identical. It's just that in the first example, $\sigma_c$ denotes that tanh (on the 4th line).

And in the second example, they actually write $tanh$ on a separate line. It's just syntaxis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.