I am attempting to make a LTSM RNN in python from scratch and I have completed the code for forward pass but I am struggling to find a clear outline of the equations I need to calculate to get the gradients using back-propagation. Is there any straightforward resource that I can learn these equations from and how to implement them(as my programming skills are limited)? Thanks for any help,

The notation for my code is that; hu is the update gate, hf is the forget gate and ho is the output gate.

def forward(inp,target):
    loss = 0
    c_temp,c,x,a,y,prob = {},{},{},{},{},{}
    c_old = {}
    c[-1] = np.zeros((hidden_size,1))
    a[-1] = np.zeros((hidden_size,1))
    for t in range(len(inp)):
        x[t] = np.zeros((vocab_size,1))
        x[t][inp[t]] = 1
        X = np.concatenate((x[t],a[t-1]))
        c_temp[t] = tanh(wc @ X  + bc)
        hf[t] = sigmoid( wf @ X + bf)
        hu[t] = sigmoid( wu @ X + bu)
        ho[t] = sigmoid(wo @ X + bo)

        c[t] = hu*c_temp[t] + hf * c[t-1]
        a[t] = ho * tanh(c[t])

        y[t] = wy @ a[t]  + by
        prob[t] = softmax(y[t])
        loss += loss(prob[t],target[t])

Here is a good tutorial on LSTM from scratch with forward and backward pass.

  • $\begingroup$ Would I be right in assuming that when you update parameters you update for example the 'weight1' with the derivative of the error with respect to weight1? $\endgroup$ – treutm Dec 22 '18 at 21:25
  • $\begingroup$ Yes, that's correct $\endgroup$ – Antonio Jurić Dec 22 '18 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.