# Practical Experiments on Self-Attention Mechanisms: QQ^T vs. QK^T

I'm currently exploring the self-attention mechanism used in models like Transformers, and I have a question about the necessity of using a separate key matrix (K) instead of just using the query matrix (Q) twice, resulting in QQ^T instead of QK^T. Despite reading several materials and arguments, I'm not fully convinced why models cannot manage to use QQ^T for self-attention.

I'm asking to seek practical experiments or studies comparing the performance of QQ^T self-attention with the traditional QK^T approach. If such experiment exist, I'm particularly interested in any insights or results that could illustrate the impact of using QQ^T on model performance and learning dynamics.

QQᵀ only carries half the information of QKᵀ:

import numpy as np
Q = np.random.rand(3,3)
[email protected]


gives:

array([[0.50650823, 0.24334603, 0.55175925],
[0.24334603, 0.15435909, 0.27704719],
[0.55175925, 0.27704719, 0.60509267]])


Notice how it is symmetrical around the diagonal. (See here for a mathematical explanation)

I wanted to know the answer to your question too, so a while back (gosh, where have the past three years gone?!) I did some experiments; it is easy enough to hack a transformer to try it, and it does work. It saved some weights, but performed worse.

However even when I adjusted the model dimension so it used about the same number of weights, I couldn't get it to be better. Similarly with experiments to replace V. Dropping the output projection matrix (which is a quarter of the attention weights in a transformer model but gets overlooked in papers) was more interesting, but again I was never better than about breakeven when comparing total model weights against performance.

(These experiments were on the encoder side of an encoder-decoder NMT model; but I don't think the conclusions will be different for encoder-only or decoder-only transformers. Model dimensions ranged from 128 to 2048, and training was on a relatively small corpus of parallel sentences.)

P.S. Another way I trained my intuition was using Sage to make 2x2 matrices for Q,K,V,P and see what they were doing, then scaling up for more tokens and/or more hidden dimensions. (The software breaks down above dim 4, and 4 tokens, but then so does the human mind...)

• Thank you for the answer. I like how "QQ Trend" matches the question : ) Commented May 9 at 21:19