By just using one-hot encoding you get an output in range 0..1, which is not normal distributed, as it is expected by the first linear layer with
uniform(-1/sqrt(in_features), 1/sqrt(in_features)) initialization, so you might get longer training time and worse generalization due to input distribution and weight initialization.
On the other hand, why you should use one-hot, when you can actually just add a linear layer without bias term? This is an embedding layer, that is normal-initialized with mean 0 and std 1. By getting something at some index you get uniform-shaped activations, unlike one-hot. Embeddings layer also can be used to reduce redundant dimensionality explosion caused by one-hot encoding. You may want consider freezing weights of embeddings layer, as you don't gain much from two linear layers (input x -> embeddings -> linear) without non-linearity between embeddings layer and linear layer.
And what I can say about positional encoding in a feedforward neural network, which is BERT, GPT and BARD.
Take a look how positional encoding is encoding your inputs:
x = x + pe
By this logic you can pre-generate
pe (positional encoding) tensor out of random beta U-shaped distribution. You cannot put many features into one feature by just summing your token embedding with position embedding and with segment embedding. That's why even with segment embeddings BERT needed separation token.
Concluding, positional information in feedforward neural network is carried by the neuron itself. If you just throw out positional encoding from BERT architecture you would not lose in performance.
To prove it, there is two important papers I have just discovered:
But where positional emdebbings might work and improve performance? Of course in convolutional networks, where you have kernels, not neurons. Positional embeddings might get injected as a channel to the input.