I'm in the middle of learning about Transformer layers, and I feel like I've got enough of the general idea behind them to be dangerous. I'm designing a neural network and my team would like to include them, but we're unsure how to proceed with the encoded sequences and what the right way to plug the them into the next layer in the model would be. We would like to process it such that we can plug the encoded sequence into a FC layer immediately after the Transformer Encoder.
If we just use a batch size of 1, for the sake of the argument, our encoded sequence output after being processed by the Transformer Encoder has shape tuple of (L,E), where L is the input sequence length and E is the embedded dimension size. I've seen some vague description of using some max/avg/conv1d pooling on the Encoded sequence, but nothing super clear about what that means. If I'm following this correctly, would I apply the max/avg/1conv1d pooling such that the pooling result gives me an resulting vector with shape tuple (E,), or would I pool along the other dimension?