1
$\begingroup$

https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77

I read the blog above. It visualizes that different color/head has different attention of words.

Based on my understanding, the code implementation of each head is almost the same.

$\endgroup$
0
$\begingroup$

After re-read the code, I am wrong. Each head is different part of the tensor's last dim.

Each head is to split the last dim of the tensor.

For example, head number is 12 and last dim of the tensor should be 12*N.

Each value of the tensor's the last dim represent a feature for the tensor.

But it seems the training could let different heads be different representations. This is amazing.

Think about CNN. CNN also has a lot of kernels. And the info of kernels may have some redundancy. As the init values of kernels are different, the kernels also learn some different info.

Then watch the figure in the link again, BERT's attentions also have some redundancy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.