I was implementing the transformer architecture in tensorflow.
I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline
They implement the positional encoding in this way:
angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
However in the paper i is not divided by 2 (i//2), is this a bug? , or why is the reason to make this operation?
thanks