2
$\begingroup$

I'm currently exploring the self-attention mechanism used in models like Transformers, and I have a question about the necessity of using a separate key matrix (K) instead of just using the query matrix (Q) twice, resulting in QQ^T instead of QK^T. Despite reading several materials and arguments, I'm not fully convinced why models cannot manage to use QQ^T for self-attention.

I'm asking to seek practical experiments or studies comparing the performance of QQ^T self-attention with the traditional QK^T approach. If such experiment exist, I'm particularly interested in any insights or results that could illustrate the impact of using QQ^T on model performance and learning dynamics.

$\endgroup$

1 Answer 1

2
$\begingroup$

QQᵀ only carries half the information of QKᵀ:

import numpy as np
Q = np.random.rand(3,3)
[email protected]

gives:

array([[0.50650823, 0.24334603, 0.55175925],
       [0.24334603, 0.15435909, 0.27704719],
       [0.55175925, 0.27704719, 0.60509267]])

Notice how it is symmetrical around the diagonal. (See here for a mathematical explanation)

I wanted to know the answer to your question too, so a while back (gosh, where have the past three years gone?!) I did some experiments; it is easy enough to hack a transformer to try it, and it does work. It saved some weights, but performed worse.

However even when I adjusted the model dimension so it used about the same number of weights, I couldn't get it to be better. Similarly with experiments to replace V. Dropping the output projection matrix (which is a quarter of the attention weights in a transformer model but gets overlooked in papers) was more interesting, but again I was never better than about breakeven when comparing total model weights against performance.

(These experiments were on the encoder side of an encoder-decoder NMT model; but I don't think the conclusions will be different for encoder-only or decoder-only transformers. Model dimensions ranged from 128 to 2048, and training was on a relatively small corpus of parallel sentences.)

P.S. Another way I trained my intuition was using Sage to make 2x2 matrices for Q,K,V,P and see what they were doing, then scaling up for more tokens and/or more hidden dimensions. (The software breaks down above dim 4, and 4 tokens, but then so does the human mind...)

$\endgroup$
1
  • $\begingroup$ Thank you for the answer. I like how "QQ Trend" matches the question : ) $\endgroup$
    – Peyman
    Commented May 9 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.