1
$\begingroup$

I am writing a paper in Natural Language Processing (NLP), and I just have a quick question about terminology.

In language models like Transformers, "token" refers to individual word in a text sequence, whereas there is a special term "token embedding" to refer to the embedding that results after token gets passed through the initial embedding layer.

Would it be problematic if I just refer a "token embedding" as a "token"?

(e.g. "interaction between hidden embeddings and token embeddings" ---> "interaction between hidden embeddings and tokens")

I am trying to accommodate the different terminologies, but my sentences are getting really wordy...

Thank you,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

At least to me, it would sound strange, as I would understand tokens as the discrete textual units they are, not their assigned vectors.

I would suggest that you don't try to force nonstandard simplifications. Just say what you want, in a technically accurate way, preferably using short sentences to avoid making it difficult for the reader to follow your discourse.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Completely agree. A token and its "representation" are different things. If you want to make your sentences less wordy, you can come up.with your own acronyms, i.e. TE (token embeddings). But otherwise, you should never lose accuracy to make reading easier. $\endgroup$
    – Valentin Calomme
    Commented Jun 7, 2020 at 21:37
  • $\begingroup$ Thank you for your kind feedback. I appreciate it. $\endgroup$
    – HDB
    Commented Jun 7, 2020 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.