If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand. I don't understand the argument that varying length inputs are taken care of by sharing weights as stated in many StackExchange answers like https://stats.stackexchange.com/questions/221513/why-are-the-weights-of-rnn-lstm-networks-shared-across-time or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce. If in the architecture below I use different $W_{e}^{(t)}$ for each word at time $t$ then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has the same size). And if we similarly take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have the same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previously hidden state vector of let's say dimension $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using the same $W_{h}$ and $W_{e}$ solve the problem of variable input sequence lengths?

Also, I know that in standard RNNs like below hidden state kind of store the context from previous time steps so what is the interpretation of $W_{h}$ here?

enter image description here

  • $\begingroup$ Can an RNN (by definition )simply not share weights between time-shifted inputs? At some point or another weights have to be shared simply because of the loops or time delays involved $\endgroup$
    – Nikos M.
    Jun 19 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.