1
$\begingroup$

videopic

from https://www.youtube.com/watch?v=TrdevFK_am4 that explains the paper titled, "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE"

compare that to the architecture shown here https://jalammar.github.io/illustrated-transformer/

So the ViT has a simpler architecture? It seems like the output of the encoder is the input to the MLP for the classification tasks.

Also I was referred to this repo https://github.com/lucidrains/vit-pytorch for learning purposes.

Are there any other ones I should know about?

I took a computational photography class at GaTech OMSCS (my specialization says robotics and computational perception) but that was in 2019 so I need to do some catching up, not to mention computer vision is different from photography.

Please feel free to link to additional resources which I should be going through.

$\endgroup$

2 Answers 2

2
$\begingroup$

It depends on the task you want to perform. The goal here is to find a way to represent your image as a sequence of embedding vectors representing each patch of the image Once you have obtained an encoded representation of your input image, you can

  • perform image classification adding a classification head (i.e a few linear layers, as shown in the picture). Here's an example image classification with ViT on Huggingface
  • generate new sequences of images or text (e.g adding a GPT decoder which takes the sequence of hidden states and outputs a sequence of tokens for tasks like image captioning, visual question answering ...). Here's an example of image captioning on Huggingface. VQA is quite similar, only requiring to pass to the decoder a representation of the image and the question as text.
$\endgroup$
2
  • $\begingroup$ I have some experience with transfer learning ResNet50 for classification but can you post some links to tutorials for both points you mention if there are any? Especially number 2. $\endgroup$ May 7, 2023 at 2:28
  • $\begingroup$ I've added an example for both cases. Anyways, the main message is that once you have found a clever way to represent an image as a sequence of encoded patches, then you can apply any Sequence Classification / Seq2Seq task you like, just like with text. $\endgroup$
    – Ciodar
    May 7, 2023 at 12:32
1
$\begingroup$

I prompted ChatGPT and received a good response including a link to https://ai.googleblog.com/2020/12/transformers-for-image-recognition-at.html

Prompt:

Please link to some implementation examples and tutorials, including Hugging Face, for the paper titled AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE so I can learn more about it.

Response:

Here are some example implementations and tutorials for the paper "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE":

  1. Hugging Face's official implementation of the ViT model in PyTorch: https://github.com/huggingface/transformers/tree/master/examples/vision

  2. TensorFlow's official implementation of the ViT model: https://github.com/google-research/vision_transformer

  3. A tutorial on how to use the ViT model in TensorFlow: https://keras.io/examples/vision/image_classification_with_vision_transformer/

  4. A tutorial on using the ViT model to classify images in PyTorch: https://towardsdatascience.com/transformers-for-image-recognition-61527a2fed19

  5. A blog post discussing the paper and its implications: https://ai.googleblog.com/2020/12/transformers-for-image-recognition-at.html

  6. A YouTube video explaining the paper and its key ideas: https://www.youtube.com/watch?v=TrdevFK_am4

That was my second prompt because I knew Hugging Face had one which was missing from the first prompt's response:

  1. Official Implementation by authors: https://github.com/google-research/vision_transformer
  2. PyTorch implementation by Ross Wightman: https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer
  3. TensorFlow implementation by Piotr Dollar: https://github.com/pdollar/coco/tree/master/PythonAPI/pycocotools
  4. Keras implementation by Ali Salloum: https://github.com/Alsaoudi-Salloum/ViT-Keras
  5. Jax implementation by Ryan Reece: https://github.com/google/flax/tree/main/examples/vision_transformer
  6. TensorFlow implementation with added Efficient Attention Module by GitHub user Vignesh Murali: https://github.com/Holmesalbatross/ea-myelonet/blob/main/eamodel.py.

From the Google blog you can watch an animation that illustrates the simplicity of the model:

animation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.