1
$\begingroup$

I have been trying to understand facebook's Detection transformer(DeTr) paper.

Architecture

enter image description here Most of the explanation about the architecture is straightforward.
I don't especially understand the concept of object queries.

Details

In the paper,

A transformer decoder then takes as input a small fixed number of learned positional embeddings, which we call object queries, and additionally attends to the encoder output.

There are 2 kinds of positional encodings in our model: spatial positional encodings(fixed) and output positional encodings(Object queries).

FYI:
Learned positional encodings and output positional encodings are same.

Questions

  1. What are these "learned" positional encodings(object queries)?
  2. how are they different from spatial positional encodings?
  3. When are they exactly learned?
$\endgroup$

1 Answer 1

0
$\begingroup$

Transformers typically convert a sequence of vectors of a fixed length, d, to an output sequence of vectors of the same length.

How you arrive at the sequence of vectors to feed into the transformer depends on the problem you're trying to solve.

  • For language translation , you embed your word into a d dimensional embedding.
  • For image recognition you learn a mapping from fixed size patches of the image to a d dimensional vector.

But sometimes, you do not have an input vector to provide to the model. Example cases taken from the BERT include

  • When the first output is a special case, the classification of the whole sequence. This is the first [CLS] token.
  • When the input token is masked, it's replaced with a [MASK] token.
  • When you want to show the separation between two sentences, you use a [SEP] token.

In case of the DeTr paper, the decoder stage does not have any inputs. So, they simply learn a d dimensional embedding at each position, and call these object queries, since these are used to query the learned encoding of the input image to generate the outputs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.