# Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

I was implementing the transformer architecture in tensorflow.

I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline

They implement the positional encoding in this way:

angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))

However in the paper i is not divided by 2 (i//2), is this a bug? , or why is the reason to make this operation?

thanks

It's not a bug, although they added some confusion with this trick. They should better call their argument $$j$$ instead of $$i$$, cos what they actually do is they take all values $$0 \leq j \leq d_{model} - 1$$ and compute $$PE(pos, j)$$. $$j$$ сan be either even or odd, but in the right side of the equation it's even, that's why they compute i//2 and multiply back by 2.

I think you should understand the equation like this: When the dimension(i) is even, we calculate the position encoding like

But when the dimension is odd, we should make the calculation using the former index which means when we calculate the value of dimension 3， we should take 2 into the equation as the parameter i.

I recommend to read the simple code in a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1. The author realize the postion encoding as :

import numpy as np
import matplotlib.pyplot as plt

def getPositionEncoding(seq_len, d, n=10000):
P = np.zeros((seq_len, d))
for k in range(seq_len):
for i in np.arange(int(d/2)):
denominator = np.power(n, 2*i/d)
P[k, 2*i] = np.sin(k/denominator)
P[k, 2*i+1] = np.cos(k/denominator)
return P

P = getPositionEncoding(seq_len=4, d=4, n=100)
print(P)

in the second for loop, the index range is [0,int(d/2)] For example， when we have dimension 4 [0,1,2,3], the index range is [0,2] which is [0,1,2]. When the i is 2, we calculate the values for dimension 2 and 3 and they have the same denominator. This is consistant with what I am saying when the dimension is odd, we should make the calculation using the former index

This is my understanding way