0
$\begingroup$

I'm reading the paper Attention is all you need here and came along the following sentence:

"Recurrent models typically factor computation along the symbol positions of the input and output sequences."

I think I understand recurrent networks pretty well but I don't understand what "factor computation" means in this context, and what "along symbol positions". Are they simply saying that the computation occurs in a sequentially through the net or is it something else? If so, why would they use that specific wording?

The full quote is:

"Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states $h_t$, as a function of the previous hidden state $h_{t−1}$ and the input for position $t$. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains."

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.