Keras multihead attention if used as single head num_heads=1, then how is it different than Keras Attention ?
Also, Is multihead attention by default self-attention type?
$\begingroup$
$\endgroup$
Add a comment
|
$\begingroup$
$\endgroup$
2
The Keras attention layer is a Luong attention block of type dot-product. Optionally, you can specify the layer to have a learnable scaling factor, with use_scale=True.
A single-head attention block from the Transformer model is also a dot-product, but scaled to the fixed dimension of the embedding ($\frac{1}{\sqrt{d_k}}$).
Therefore, their only difference is the scaling factor, which is learned in the case of the Keras attention layer (if enabled), while it is fixed in the case of the Transformer attention.
-
$\begingroup$ Thanks noe. That answered 2nd question. Wondering if 1st question is also answered - Keras multihead attention if used as single head num_heads=1, then how is it different than Keras Attention ? (here looking for difference between 2 implementations, and does the second can be scaled down to 1st?) $\endgroup$ Jun 16 at 19:29
-
$\begingroup$ As I commented in the answer, their only difference is that the scale is a learnable parameter in the case of the Keras Attention layer and fixed in the case of the single-headed Transformer attention. I am not sure what you mean with "does the second can be scaled down to 1st?". Do you mean how to implement a Transformer attention based on the Keras Attention layer? If so, you can simply have
use_scale=Falseand apply the fixed scaling manually. $\endgroup$– noeJun 16 at 22:54