Here is some code that initialises weights for an RNN:

u = np.random.randn(hidden_size, input_size) * alpha
v = np.random.randn(hidden_size, hidden_size) * alpha
w = np.random.randn(output_size, hidden_size) * alpha

I have some questions about the above. First, I understand that random.randn generates numbers based around the standard normal distribution. How does doing so contribute to efficiency, and why is it favourable? Similarly, why are the weights then multiplied by the learning rate? Finally where I am most confused, why are the parameters for u hidden_size then input_size, rather than the other way around (same for w)? I understand that changing this order messes with the matrix broadcasting further down the line. But there a strategy behind this particular order?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.