New answers tagged


(Answering just the first part of the question, based on my own understanding.) Multi-head attention is a randomly-initialized multi-dimensional indexing system where different heads focus on different variations of the indexed token instance (token = initially e.g. a word or a part of a word) with respect to its containing sequence/context. Such encoded ...


Update for anyone googling this in 2021: Keras has implemented a MultiHead attention layer. If key, query, and value are the same, this is self-attention.


Character-level models are only rarely better than subwords, not even in situations where you would naturally expect it (cf. recent papers When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation, Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems). The biggest gains ...

Top 50 recent answers are included