I am writing a paper in Natural Language Processing (NLP), and I just have a quick question about terminology.

In language models like Transformers, "token" refers to individual word in a text sequence, whereas there is a special term "token embedding" to refer to the embedding that results after token gets passed through the initial embedding layer.

Would it be problematic if I just refer a "token embedding" as a "token"?

(e.g. "interaction between hidden embeddings and token embeddings" ---> "interaction between hidden embeddings and tokens")

I am trying to accommodate the different terminologies, but my sentences are getting really wordy...

Thank you,


At least to me, it would sound strange, as I would understand tokens as the discrete textual units they are, not their assigned vectors.

I would suggest that you don't try to force nonstandard simplifications. Just say what you want, in a technically accurate way, preferably using short sentences to avoid making it difficult for the reader to follow your discourse.

  • $\begingroup$ Completely agree. A token and its "representation" are different things. If you want to make your sentences less wordy, you can come up.with your own acronyms, i.e. TE (token embeddings). But otherwise, you should never lose accuracy to make reading easier. $\endgroup$ – Valentin Calomme Jun 7 '20 at 21:37
  • $\begingroup$ Thank you for your kind feedback. I appreciate it. $\endgroup$ – HDB Jun 7 '20 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.