Here is some code that initialises weights for an RNN:
u = np.random.randn(hidden_size, input_size) * alpha v = np.random.randn(hidden_size, hidden_size) * alpha w = np.random.randn(output_size, hidden_size) * alpha
I have some questions about the above. First, I understand that
random.randn generates numbers based around the standard normal distribution. How does doing so contribute to efficiency, and why is it favourable? Similarly, why are the weights then multiplied by the learning rate? Finally where I am most confused, why are the parameters for
input_size, rather than the other way around (same for
w)? I understand that changing this order messes with the matrix broadcasting further down the line. But there a strategy behind this particular order?