What is “position” in CNN (im2latex) for Positional Encoding?

Question

I'm trying to build a model that maps images of math formulas into LaTeX markup.

I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder architecture that solves this problem.

In encoder after the work of CNN it's proposed (page 3) to add following signals to generated feature maps "in order to retain the spatial locality information":

$$PE(x,y,2i)=sin(x/10000^{4i/D})$$ $$PE(x,y,2i+1)=cos(x/10000^{4i/D})$$ $$PE(x,y,2j+D/2)=sin(y/10000^{4j/D})$$ $$PE(x,y,2j+1+D/2)=cos(y/10000^{4j/D})$$

Here the authors refer to positional encoding part of Transformer model "tailoring the 1-D positional encoding technique to 2-D".

In original Transformer model (from article "Attention Is All You Need") the position encoding is calculated like this:

Where "pos" is just an index number of word in given sentence (correct me if I'm wrong).

And here comes the question: what is "x" and "y" in first formula?

Answer 1

In the original transformer model, one input 'pos' was required as input was 1 dimensional (words). With encoder being CNN, the input feature vector will be 2D, so x and y are just the indexes of an 2D feature vector. By using those indexes, PE functions can produce those positional encoder images that you can find in the paper.