Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/55158766?noredirect=1#comment97066160_55158766

I am coming from Google BERT context (Bidirectional Encoder representations from Transformers). I have gone through architecture and codes. People say this is bidirectional by nature. To make it unidirectional attention some mask is to be applied.

Basically a transformer takes key, values and queries as input; uses encoder decoder architecture; and applies attention to these keys, queries and values. What I understood is we need to pass tokens explicitly rather than transformer understanding this by nature.

Can someone please explain what makes transformer bidirectional by nature

Answer received so far:
1. People confirmed that Transformer has Bidirectional nature, rather than an external code making it bidirectional.
2. My doubt: We are passing Q K V embeddings to transformer, to which it applies N layers of self attention using ScaledDotMatrix attention. Same thing can be done by unidirection approach as well. May I know what part I am missing in my understanding. If someone can point to code where it is getting bidirectional, it would be a great help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.