0
$\begingroup$

Positional embeddings are introduced into a transformer in order to add positional information to a word embedding.

Now, suppose we have an existing data embedding that can be for any data domain word/image. We don't have the original text/image before being encoded but the final embedding. Is it possible to extract positional embedding from an embedding or do we have to run positional embedding on the raw input (text/image) please? Is there a way around it in case we don't have the original raw input but only its embedding?

$\endgroup$

2 Answers 2

1
$\begingroup$

Not possible in a typical Transformer AFAIK. Most neural networks are non-invertible mappings, so you cannot guaranteed reconstruct their inputs from their hidden layers. For transformers specifically, the positional embedding is added to the token embedding before being fed into self-attention, so there not an easy way to disentangle these that I'm aware of.

$\endgroup$
1
$\begingroup$

In the Transformer architecture, the positional embedding is added to the data embedding vectors so, if you had just the embedded data and kept the ordering between data vectors, you could just pass it through the normal transformer:

enter image description here

$\endgroup$
5
  • $\begingroup$ No the data embedded before was not aware of the position. So I guess I can not add positional embedding without the raw data then? But I have something interesting though in the embedded data, the data has a groups of embeddings for per cluster (group of greetings sentences), (group of business sentences), ... etc. $[embed_1, embed_2, ..., embed_n]$ (greetings) $\cdots$ $[embed_1, embed_2, ..., embed_n]$ (business) where each $embed_i \in \mathbb{R}^{100}$. $\endgroup$
    – Avv
    Jan 7, 2023 at 15:58
  • 1
    $\begingroup$ But is there any meaningful ordering in the raw data? $\endgroup$
    – noe
    Jan 7, 2023 at 16:42
  • $\begingroup$ I think yes due to the format of the raw data (Tree representation of text embedded). Unfortunately, the whole text is embedded into one vector of size 100. I was thinking of splitting the 100 into 4 vectors each of size 25 and then try to add positional encoding to the, but not sure if this is wise though? $\endgroup$
    – Avv
    Jan 8, 2023 at 0:32
  • 1
    $\begingroup$ No theoretically grounded, but you never know if it actually works until you try ¯_(ツ)_/¯ $\endgroup$
    – noe
    Jan 8, 2023 at 8:01
  • $\begingroup$ Thank you. I was thinking that maybe the tree structure of the text before it was embedded into a 100-dimensional vector has some sort of order! $\endgroup$
    – Avv
    Jan 8, 2023 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.