0
$\begingroup$

I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example context should be more important than grammar or syntax and positioning should have another weight. How could I express it?

Thank You in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

In the context of word embeddings, it's not typical to train words or sentences in two layers with different importance. Word embeddings are usually trained in a single layer where each dimension captures different aspects of the word's meaning, including its syntax, semantics, and context.

However, what you're describing sounds more like a task for a more complex model, such as a transformer model like BERT or a recurrent neural network (RNN) like LSTM. These models can capture both the syntax and semantics of words in a sentence, as well as their position.

If you want to emphasize certain aspects more than others, you could potentially do this during the training process by using different loss functions or by weighting the errors differently. For example, you could assign a higher weight to errors made on the context of the word compared to its syntax or position.

Improve this answer
$\endgroup$
8
  • $\begingroup$ First thank You for the answer. I would like to point out that I m new on the filed, I m more linguist. So may be my questions may be very easy to answer but I need answers for my research. I could use BERT to train a contextualised word embedding model. Bert takes only subwords? not whole words? do I have to lemmatise? Yes I want to train it in two layers eg on sentence level and on phrase level. Then may be concatenate or find the dot product? would that be useful? Do you have an article to recommend? $\endgroup$
    – Christina Valavani
    Commented Jul 18, 2023 at 9:24
  • $\begingroup$ BERT is trained on subwords, but it doesn't mean that it cannot handle whole word. It's like this: Let's say your name: "Christina". If there are embeddings available for this word, then the BERT will give you those embeddings. But there is no embedding for this word, then BERT will separate this into two subwords like Chris and ##ina. $\endgroup$
    – Harshad Patil
    Commented Jul 18, 2023 at 9:49
  • $\begingroup$ Then it will again look into its vocab and then look for embeddings for these two subwords. If available, then it will provide you the embeddings for these two words, or it will keep dividing to the character level. Also, you don't need to lemmatize them because it will lose the context (right?) $\endgroup$
    – Harshad Patil
    Commented Jul 18, 2023 at 9:49
  • $\begingroup$ To understand the BERT model, please go through the following link: jalammar.github.io/illustrated-transformer $\endgroup$
    – Harshad Patil
    Commented Jul 18, 2023 at 9:50
  • $\begingroup$ Can I do BERT on parsed data? For Bert do I need pre embedded words? Or it takes as input just words? $\endgroup$
    – Christina Valavani
    Commented Jul 18, 2023 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.