A few years ago, I understood the classical MLP neural network much better when I wrote an implementation from scratch (using only Python + Numpy, without using tensorflow). Now I'd like to do the same for recurrent neural networks.

For a standard MLP NN with dense layers, the forward-propagation can be summarized by:

def predict(x0):
    x = x0
    for i in range(numlayers-1):
        y = dot(W[i], x) + B[i]     # W[i] is a weight matrix, B[i] the biases 
        x = activation[i](y)
    return x

For a given single layer, the idea is just:

output_vector = activation(W[i] * input_vector + B[i])

What's the equivalent for a simple RNN layer, eg. SimpleRNN ?

More precisely, let's take an example of a RNN layer like this:
Input shape: (None, 250, 32)
Output shape: (None, 100)
Given an input x of shape (250, 32), with which pseudo-code can I generate the output y of shape (100,), of course by using weights, etc.?


2 Answers 2


Simple RNN Cells follow this pattern:

Given the following data:
    input data:         X
    weights:            wx
    recursive weights:  wRec

Initialize initial hidden state to 0

For each state, one by one:
    Update new hidden state as: (Input data * weights) + (Hidden state + recursive weights)

In Python code:

def compute_states(X, wx, wRec):
    Unfold the network and compute all state activations 
    given the input X, input weights (wx), and recursive weights 
    (wRec). Return the state activations in a matrix, the last 
    column S[:,-1] contains the final activations.
    # Initialise a matrix that holds all states for all input sequences.
    # The initial state s_0 is set to 0, each of the others will depend from the previous.
    S = np.zeros((X.shape[0], X.shape[1]+1))

    # Compute each state k from the previous state ( S[:,k] ) and current input ( X[:,k] ), 
    # by use of the input weights (wx) and recursive weights (wRec).
    for k in range(0, X.shape[1]):
        S[:,k+1] = (X[:,k] * wx) + (S[:,k] * wRec)

    return S

This is a slightly more clear version of the code I found here.

Is this helpful for you?

  • $\begingroup$ LSTMs are much more complicated layers. The simple RNN layer has its SimpleRNN() class, you should substitute that to LSTM(). And yes, the function above is a forward pass that is calculated at each training iteration. $\endgroup$
    – Leevo
    Mar 7, 2020 at 8:08
  • $\begingroup$ Sorry Leevo, I did a mistake, I meant this: model = Sequential() model.add(Embedding(5000, 32, input_length=250)) model.add(SimpleRNN(100)) model.add(Dense(1, activation='sigmoid')). What's the difference between SimpleRNNCell and SimpleRNN? $\endgroup$
    – Basj
    Mar 7, 2020 at 12:01
  • $\begingroup$ I think you need to apply an activiation function inside compute_states. $\endgroup$
    – Basj
    Mar 7, 2020 at 12:02
  • $\begingroup$ I also asked the same question once here, it was related to LSTM() and LSTMCell() objects, but it's the same issue. One is the abstract class of the other. In practice, you always use the SimpleRNN() one, without the "Cell". Easy ;) $\endgroup$
    – Leevo
    Mar 10, 2020 at 14:09
  • $\begingroup$ Sorry, I'm not sure I understood your comment on compute_states. Could you elaborate on that? $\endgroup$
    – Leevo
    Mar 10, 2020 at 14:10

In other words, what does the forward pass of a RNN look like. You read about using the inputs plus values from the previous node (here it will be prev_s) First initialise the weights, than perform the foreward pass. I highlighted what you was looking for.

U = np.random.uniform(0, 1, (hidden_dim, T))
W = np.random.uniform(0, 1, (hidden_dim, hidden_dim))
V = np.random.uniform(0, 1, (output_dim, hidden_dim))

 for i in range(Y.shape[0]):
        x, y = X[i], Y[i]

        layers = []
        prev_s = np.zeros((hidden_dim, 1))
        dU = np.zeros(U.shape)
        dV = np.zeros(V.shape)
        dW = np.zeros(W.shape)

        dU_t = np.zeros(U.shape)
        dV_t = np.zeros(V.shape)
        dW_t = np.zeros(W.shape)

        dU_i = np.zeros(U.shape)
        dW_i = np.zeros(W.shape)

        # forward pass
        for t in range(T):
            new_input = np.zeros(x.shape)
            new_input[t] = x[t]
            mulu = np.dot(U, new_input)
            mulw = np.dot(W, prev_s)
            add = mulw + mulu
            s = sigmoid(add)
            ***mulv = np.dot(V, s)***
            layers.append({'s':s, 'prev_s':prev_s})
            prev_s = s

So the '* *' area can be roughly translated: mulv = np.dot(V, s) are the weights multiplied with the current state. (same as before, s==input_vector) but the difference is that the s will be calculated with weights from previous output and current input i.e.

mulu = np.dot(U, new_input)
mulw = np.dot(W, prev_s)
add = mulw + mulu
s = sigmoid(add)

Thats why we have 3 initial weights in the first place.

  • $\begingroup$ Thank you. Could you add your source, for future reference? $\endgroup$
    – Basj
    Mar 6, 2020 at 21:10
  • $\begingroup$ I have coded it long time ago on kaggle. $\endgroup$
    – Noah Weber
    Mar 13, 2020 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.