I was implementing the transformer architecture in tensorflow.

I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline

They implement the positional encoding in this way:

angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))

However in the paper i is not divided by 2 (i//2), is this a bug? , or why is the reason to make this operation?

enter image description here



2 Answers 2


It's not a bug, although they added some confusion with this trick. They should better call their argument $j$ instead of $i$, cos what they actually do is they take all values $0 \leq j \leq d_{model} - 1$ and compute $PE(pos, j)$. $j$ сan be either even or odd, but in the right side of the equation it's even, that's why they compute i//2 and multiply back by 2.


I think you should understand the equation like this: When the dimension(i) is even, we calculate the position encoding like enter image description here

But when the dimension is odd, we should make the calculation using the former index which means when we calculate the value of dimension 3, we should take 2 into the equation as the parameter i.

I recommend to read the simple code in a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1. The author realize the postion encoding as :

import numpy as np
import matplotlib.pyplot as plt
def getPositionEncoding(seq_len, d, n=10000):
    P = np.zeros((seq_len, d))
    for k in range(seq_len):
        for i in np.arange(int(d/2)):
            denominator = np.power(n, 2*i/d)
            P[k, 2*i] = np.sin(k/denominator)
            P[k, 2*i+1] = np.cos(k/denominator)
    return P
P = getPositionEncoding(seq_len=4, d=4, n=100)

in the second for loop, the index range is [0,int(d/2)] For example, when we have dimension 4 [0,1,2,3], the index range is [0,2] which is [0,1,2]. When the i is 2, we calculate the values for dimension 2 and 3 and they have the same denominator. This is consistant with what I am saying when the dimension is odd, we should make the calculation using the former index

This is my understanding way


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.