Skip to main content
added 303 characters in body
Source Link
noe
  • 27.5k
  • 1
  • 49
  • 80

About the first piece of code you posted:

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.


About the second piece of code you posted:

It seems to add up the subword embeddings of each word (only the last BERT layer) and concatenate each resulting vector into a tensor for the whole sentence (whose length would be the number of words).

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.

About the first piece of code you posted:

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.


About the second piece of code you posted:

It seems to add up the subword embeddings of each word (only the last BERT layer) and concatenate each resulting vector into a tensor for the whole sentence (whose length would be the number of words).

Source Link
noe
  • 27.5k
  • 1
  • 49
  • 80

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.