0
$\begingroup$

I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT.

To the best of my understanding this token is used to gather knowledge of the entire class, and is then solely used to predict the class of the image. My question is — why does this token exist as input in all the transformer blocks and is treated the same as the word / patches tokens?

Treating the class token like the rest of the tokens means other tokens can attend to it. I'd expect that the class token will be able to attend other tokens while they could not attend it.

Also, specifically in ViT, why does the class token receive positional encodings? It represents the entire class and thus doesn't have any specific location.

Thanks!

$\endgroup$
0
$\begingroup$

My question is — why does this token exist as input in all the transformer blocks and is treated the same as the word / patches tokens?

The CLASS token exists as input with a learnable embedding, prepended with the input patch embeddings and all of these are given as input to the first transformer layer. The CLASS token gathers information from all the patches using Multihead Self Attention (MSA). It is basically treated the same as patch tokens but at the end when doing classification, only the hidden output from the CLASS token are used as input to the classification layer.

Treating the class token like the rest of the tokens means other tokens can attend to it. I'd expect that the class token will be able to attend other tokens while they could not attend it.

I think that would help training all the weights during back-prop, including the weights of the embedding layer of the input patches.

Also, specifically in ViT, why does the class token receive positional encodings? It represents the entire class and thus doesn't have any specific location.

The positional encoding would tell that this is the first element of the sequence. Would help if you have a bi-directional transformer model like BERT.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.