0
$\begingroup$

I have a question about the score function of ALiBi (Attention with Linear Biases), which is a positional encoding method introduced by the following paper:

TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR BIASES ENABLES INPUT LENGTH EXTRAPOLATION

According to Figure 3 in the paper, the score function form is a lower triangular matrix.

If in the masked multi-head attention in the Transformer model (e.g., in decoder layer), it is no problem.

On the contrary, I'd like to know about its use in a general format, in other words, in normal multi-head attention (e.g. encoder layer).

Is it also a lower triangular matrix? If not, what form does the linear bias term have?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.