My question is — why does this token exist as input in all the
transformer blocks and is treated the same as the word / patches
The transformers, by default are sequence to sequence networks. As there is no decoder layer in ViT, then the length of input sequence (number of patches) equals the length of output sequence. So If the goal is classification, there is two choices:
- Either apply a fully connected layer on top of the transformer (which is not a good idea because then we have to fix the number of patches--which translates to input image resolution)
- Or apply the classification layer on one items of the output sequence, but which one?! The best answer here is none of them! We don't want to be biased toward any of the patches. So the best solution here is to add a dummy input, call it class token and apply the classification layer on the corresponding output item!
Treating the class token like the rest of the tokens means other
tokens can attend to it. I'd expect that the class token will be able
to attend other tokens while they could not attend it.
Not sure, but I think if other tokens can attend to class token, then they can use some intermediate information about image class in lower layers! Just a guess and it worth to test different scenarios!
Also, specifically in ViT, why does the class token receive positional
encodings? It represents the entire class and thus doesn't have any
I think the main reason is that, this way the network can distinguish a class embedding from patch embedding and treat them differently!